Customer Portal Outage

Incident Report for Yext

Postmortem

Summary

On October 29th, 2024, at approximately 3:00PM ET, the Customer Portal experienced an outage for nearly two hours. Some remaining errors remained for the subsequent 2.5 hours, and the portal was returned to an error-free state by approximately 7:45PM ET. At that point, some translations were unavailable. By 10:00PM ET, the translations were fully restored.

Root Cause

Our front-end web services depend on language translation files to serve the Customer Portal in various languages. These files are hosted on a legacy storage system, and that storage system had become desynchronized with the application servers that utilize them. This caused slowness when accessing these files. The slowness caused a cascade of issues that ultimately resulted in the Customer Portal being unable to start.

‌

To remediate this issue, the entire fleet of application servers had to be restarted in a rolling fashion. This was time-consuming, leading to the protracted downtime.

Remediation

Prior to this incident, we had already begun deprecating the legacy storage system and had migrated many workloads off it. We will redouble our efforts and prioritize migrating these translation files to the new architecture.

‌

Additionally, we have added telemetry to notify us more quickly if the legacy storage system experiences similar problems in the future.

‌

Lastly, we have refined the process for restarting our fleet of application servers so that we can recover more quickly in the worst-case scenario.

Posted Nov 06, 2024 - 11:03 EST

Resolved

This incident is resolved.
Posted Oct 29, 2024 - 23:21 EDT

Monitoring

We have remediated the errors. We will continue to monitor the situation.
Posted Oct 29, 2024 - 22:04 EDT

Update

We have identified the source of errors and addressed the underlying issue. The customer portal is operational at this time. Translations for non-English languages may be missing from the portal, we are working to restore them.
Posted Oct 29, 2024 - 19:45 EDT

Identified

We have largely remediated the customer portal issues. A small subset of errors are still being worked on towards resolution.
Posted Oct 29, 2024 - 17:11 EDT

Update

We are continuing to investigate this issue.
Posted Oct 29, 2024 - 16:50 EDT

Update

We are currently investigating reports of an outage within our customer portal.
Posted Oct 29, 2024 - 15:20 EDT

Investigating

We are investigating reports of errors in Partner Scan
Posted Oct 29, 2024 - 15:05 EDT
This incident affected: Connectors (Connectors Portal), Social (Social Portal, Social Posting), Search (Search Portal), Listings (Listings Portal), Chat (Chat Portal), Analytics (Analytics Portal), Knowledge Tags (Knowledge Tags Portal), Widgets (Widgets Portal), Pages (Pages Portal), Reviews (Reviews Portal), Knowledge Graph (Content Portal), and Customer Portal Login, Resellers Portal, Notifications Portal.