On June 9th, beginning at 21:12 EDT, Yext engineers began to observe a gradual increase in listing delivery latencies. Engineers continued monitoring closely while looking for root cause. The following day, delivery latencies had escalated beyond our normal operating ranges, and more engineers joined investigative efforts by 12:07 EDT. The cause of the delay was identified by 12:34 EDT, the mitigations were implemented by 13:26 EDT, and listing delivery latencies returned to normal levels at 13:57 EDT.
As part of our scheduled maintenance on June 9th, we conducted routine upgrades and security patching on a number of database servers. As a result of this upgrade combined with increased load, performance issues occurred that were not revealed in pre-production testing.
As a result, we have developed, tested, and deployed changes to relieve these performance issues. We have also completed modifications to our database operation procedures to add automatic safeguards that will prevent this class of issues in the future.