Increased error rates in Knowledge API
Incident Report for Yext
Postmortem

Summary

Starting on April 29th, 2021 at 7.09 p.m. ET, the Locations API (a legacy equivalent to the Knowledge API) started returning a higher than average number of errors, accounting for about 1-2% of all requests. This was remediated by our team by April 30th at 10.55 a.m. ET. The Live API as well as requests to the current Knowledge API were unaffected.

Root Cause

A series of large requests made to the Locations API resulted in increased latencies in responding to health checks, which caused restarts of the services providing the API. These restarts resulted in the errors seen by consumers of the API.

Remediation

The issue was remediated in the short term by providing increased resources to the affected services. We will further follow up by adding caching to reduce the load on these services, and improving our monitoring of these specific endpoints to catch such problems earlier.

Posted May 05, 2021 - 10:48 EDT

Resolved
This incident has been resolved.
Posted Apr 30, 2021 - 11:42 EDT
Update
We are continuing to monitor for any further issues.
Posted Apr 30, 2021 - 10:57 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 30, 2021 - 10:56 EDT
Investigating
We are currently investigating a higher than normal rate of errors in our Knowledge API. Live API is unaffected.
Posted Apr 30, 2021 - 09:21 EDT
This incident affected: Platform API (Knowledge and Administrative).