On the morning of October 10th, US Eastern time, Yext engineers began tracking increasing error rates in our Pages Serving traffic. Users encountering errors were presented with an error page from our CDN provider. We immediately escalated the issue to our CDN provider. At 5:11 PM ET, our CDN provider deployed a fix in their systems, and we observed that the error rates returned to normal. The errors affected less than 1% of all Pages traffic.
A change made by our CDN provider caused a small percentage of connection attempts to Yext servers to time out.
A review of our telemetry showed that this class of errors first appeared at much lower rates in the two days leading up to this incident. Because these error rates were within our service objectives, they did not trigger our standard suite of alerts. We plan to reduce noise in our error rates and increase the sensitivity of our monitoring to provide earlier warnings for future incidents.