Listings not being updated
Incident Report for Yext
Postmortem

Summary

On Tuesday, September 14th, 2021, starting at 5:55 p.m. ET, updates to listings via the Knowledge Network began experiencing extended delays, with no updates being sent until 6:41 p.m. ET. Delays continued for a small number of listings until approximately 11:00 a.m. ET on September 15th, when normal service resumed.

Root Cause

A backend service responsible for queueing updates for listings failed during a period of high load, and recovery operations resulted in some updates being lost. These updates were recovered by re-publishing the listings for any Entities that had been edited during the time of the outage.

Remediation

The backend service that experienced the outage has had additional resources applied to handle greater load, and we plan to update monitoring and management of our pipeline to better detect and control such load in future.

We will also be updating our procedures for recovering updates to

Posted Sep 17, 2021 - 09:26 EDT

Resolved
This incident has been resolved.
Posted Sep 15, 2021 - 11:21 EDT
Update
The majority of listings are now up to date, with exceptions across a small number of publishers. We're continuing to monitor and will update as the recovery progresses.
Posted Sep 15, 2021 - 10:48 EDT
Update
The remaining delayed updates are being processed and restored. We expect all listings to be up to date within the next 2 hours.
Posted Sep 15, 2021 - 00:09 EDT
Update
New updates are being processed normally, but we are still working to recover and apply some of the delayed updates. We will continue to monitor and post updates.
Posted Sep 14, 2021 - 21:07 EDT
Monitoring
We have remediated the issue, and are processing the delayed data. We will continue to monitor the situation.
Posted Sep 14, 2021 - 20:00 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 14, 2021 - 18:56 EDT
Investigating
We are currently investigating an issue updating Listings.
Posted Sep 14, 2021 - 18:30 EDT
This incident affected: Knowledge Network.