On Friday, February 17th 2023, beginning at 4:30PM ET, Yext engineers identified high error rates across the Customer Portal, Admin Console, and CLI. Delays were also identified in updates to entity data across our Publisher Network, Pages, Live API, Search, and Analytics system. Updates made via platform APIs or custom ETLs may have also failed.
Pages serving, Knowledge Assistant, Sandbox environments, and our Hitchhiker site were unaffected. Services started to be restored on Saturday, February 18th, at 1:00AM ET, although some flows may initially have returned stale data. By 3:30AM ET backlogged updates were processed and all systems had caught up.
Our primary asynchronous processing system had a major hardware failure which had downstream effects on many aspects of our platform. Once the issue was identified and fixed, services were able to process backlogged updates.
We will be reviewing our asynchronous processing system, high availability configuration, and failover procedures to improve stability and speed to recovery.