Delayed Updates in Knowledge Network
Incident Report for Yext
Postmortem

Summary

On June 9th, beginning at 21:12 EDT, Yext engineers began to observe a gradual increase in listing delivery latencies. Engineers continued monitoring closely while looking for root cause. The following day, delivery latencies had escalated beyond our normal operating ranges, and more engineers joined investigative efforts by 12:07 EDT. The cause of the delay was identified by 12:34 EDT, the mitigations were implemented by 13:26 EDT, and listing delivery latencies returned to normal levels at 13:57 EDT.

Root Cause

As part of our scheduled maintenance on June 9th, we conducted routine upgrades and security patching on a number of database servers. As a result of this upgrade combined with increased load, performance issues occurred that were not revealed in pre-production testing.

As a result, we have developed, tested, and deployed changes to relieve these performance issues. We have also completed modifications to our database operation procedures to add automatic safeguards that will prevent this class of issues in the future.

Posted Jun 15, 2019 - 11:31 EDT

Resolved
This incident has been resolved.
Posted Jun 10, 2019 - 15:40 EDT
Monitoring
We have implemented fixes and listing update latencies have returned to normal. We will continue to monitor for issues.
Posted Jun 10, 2019 - 14:04 EDT
Identified
We have identified a delay in listing updates in the Knowledge Network. Updates to listings may not post immediately at this time. We will update as soon as we have more information.
Posted Jun 10, 2019 - 12:34 EDT
This incident affected: Knowledge Network.