Subscription updates paused for all accounts, and missing features in reseller accounts

Incident Report for Yext

Postmortem

Summary

On Thursday, April 4, 2024, starting around 6:00 AM ET, Yext resellers saw many features missing from their customer portal and its navigation menu when they or their customers logged in. Some resellers started seeing their features return after 10:00 AM ET, and all resellers were returned to a normal view in their customer portal by 12:30 PM ET.

Additionally, a small number of resellers saw their Listings removed from their accounts even after access to the rest of their features were restored. Restoration of these missing Listings was completed at around 8:00 PM ET on Friday, April 5.

Also starting around 6:00 AM ET on Thursday, April 4, all Yext customers saw delays in the addition or removal of services in response to subscription and unsubscription requests. Processing of service changes by non-reseller customers returned to normal at around 8:00 PM ET the same day, and all delayed changes for these customers were also applied by that time. Processing of all new service changes by reseller customers returned to normal at around 12:30 AM ET on Friday, April 5, but some delayed service changes from during the day on April 4 were not fully applied until around 8:00 PM ET on Friday, April 5.

Root Cause

Yext has a process that rechecks the subscription status of most of our reseller customers every day. A bug in that process caused the subscription-tracking system to treat a subset of those customers as no longer subscribed to Yext services. As a result, the customer portal removed access to various features for these accounts. This issue was fixed by 12:30 PM ET on April 4 when this bug was rectified.

Unlike the customer portal, which directly queries the subscription-tracking system for current subscription status, our Listings management system handles subscription changes asynchronously, reacting to messages emitted by our subscription-tracking system after every service change. Because of this bug, the subscription-tracking system emitted a very large volume of service removal notifications, but the system that relays these messages was overwhelmed by the volume and stopped processing. As a result, the service removal notifications for only a few reseller customers made it to the Listings management system, limiting the scope of this incident.

During remediation work for this incident, we prevented the system that relays service change notifications from our subscriptions system to the rest of our systems from processing any notifications until we could be confident that those notifications were no longer affected by any bugs. This caused the delays in the addition or removal of services for all customers through the course of April 4.

Part of stabilizing the system for all new service changes by Thursday evening involved disregarding most service change notifications for reseller accounts. Remediation work on April 5 involved identifying affected accounts and manually bringing the Listings system back in sync with the subscriptions system for those accounts.

Remediation

Because of the widespread effects that a bug in the process that rechecks the subscription status of most of our reseller customers can have, we plan to add additional testing stages to the deployment process of our subscription tracking system.

We are also planning to add additional operational tooling to our Listings system to reduce incident response times.

Posted Apr 15, 2024 - 13:59 EDT

Resolved

This incident is resolved.

Posted Apr 06, 2024 - 16:42 EDT

Monitoring

All new services that were added during the course of this incident have been fully activated. The system is functioning normally at this time. We will continue to monitor the situation.

Posted Apr 06, 2024 - 07:01 EDT

Update

We have now restored all of the Listings that may have gone missing or ended up in an opted out state because of this incident. We are now working to ensure that all new services that were added during the course of this incident are fully activated in the Yext system. We will update when that has been completed, expected to be later tonight (US Eastern Time).

Posted Apr 05, 2024 - 20:10 EDT

Update

We are continuing to work toward restoring lost services. At this point, we have resolved all effects on any non-reseller clients of Yext. We are making progress on restoring services for affected reseller clients; the current expectation is that the vast majority of such lost services should be restored by this evening US Eastern Time.

Posted Apr 05, 2024 - 14:56 EDT

Update

We are continuing to work toward restoring lost services.

Posted Apr 05, 2024 - 10:59 EDT

Update

We are pausing remediation work for the night (US Eastern Time). At this point, all new subscription updates should be reflected as normal. However, some reseller accounts and subaccounts may still see missing features in their Customer Portal or missing Listings or Listings in an unexpected opted out state.

We shall resume remediation work and further updates in the morning US Eastern Time.

Posted Apr 05, 2024 - 01:56 EDT

Update

We continue to work towards returning the system to normal operation.

Most updates to subscriptions are now being processed as usual, and most updates from the past day have now been applied.

Reseller subaccounts should also now see all subscribed services when logged in to their Customer Portal, though some accounts may see their Listings in an unexpected "opted out" state. We are continuing to work on restoring those to their usual state.

We shall continue to provide further updates as they become available.

Posted Apr 05, 2024 - 01:06 EDT

Update

We continue to work towards applying a fix to allow subscriptions processing to continue. We shall continue to provide updates as they become available.

Posted Apr 04, 2024 - 21:39 EDT

Update

We continue to work towards applying a fix to allow subscriptions processing to continue. We shall continue to provide updates as they become available.

Posted Apr 04, 2024 - 18:27 EDT

Update

We are currently applying a fix to allow subscriptions processing to continue. We will provide an update in 1 hour.

Posted Apr 04, 2024 - 17:12 EDT

Update

We are continuing to work toward returning the system to normal functioning.

Posted Apr 04, 2024 - 15:58 EDT

Update

We are continuing to work toward returning the system to normal functioning.

Posted Apr 04, 2024 - 13:35 EDT

Update

We have determined that other handling of subscription state changes, including signups for new services, is currently paused across all clients, not just resellers. We are continuing to work to return the system to normal functioning.

Posted Apr 04, 2024 - 12:10 EDT

Update

We have determined that access to the legacy "v1" API has not been affected. We are continuing to work on restoring all accounts to their expected state.

Posted Apr 04, 2024 - 11:11 EDT

Update

We are also investigating whether access to the legacy "v1" API has been disabled for all accounts and are working on restoring that access as well.

Posted Apr 04, 2024 - 10:41 EDT

Identified

We have identified the cause of the missing features and are now working to restore all accounts to their expected state.

Posted Apr 04, 2024 - 09:54 EDT

Investigating

We are investigating reports that many subaccounts of our resellers are missing access to features when they log in to their Yext customer portal.

Posted Apr 04, 2024 - 09:27 EDT

This incident affected: Accounts (Subscriptions).