On June 23rd, 2019 at 10:58 a.m. ET, Yext engineering was alerted of increased error rates in the Live API platform. Engineers began investigation, and saw that our third party provider was experiencing issues. Yext systems automatically failed over to a backup provider by 11:08 a.m. ET, but the service continued to experience increased latency and error rates. Mitigation efforts continued and error rates were returned to normal by 11:19 a.m. ET.
The Live API platform automatically fails over to a backup provider in the event of an outage at our primary provider. The initial alert at 10:58 a.m. ET was caused by an incident in our primary provider. Our systems automatically failed over to the backup provider, but this provider was underprovisioned and thus unable to serve production load.
We plan to add improved monitoring and processes to ensure that our backup systems are able to serve production load during provider outages.