Performance Degradation in Customer Portal and Platform API
Incident Report for Yext
Postmortem

Summary

On June 4th, 2019, at 16:41 EDT, engineering was alerted of elevated error rates and latencies in the Customer Portal and Platform API. Mitigation efforts began immediately, and the root cause was identified and resolved by 17:15 EDT, at which point service was restored.

Root Cause

A runaway process had overwhelmed a high traffic backend service, preventing it from serving requests from other systems. Engineering immediately identified and terminated the process, which allowed the backend service to recover.

Remediation

We will be upgrading the backend service and will also be implementing systems to anticipate runaway processes to minimize impact to our platform.

Posted Jun 07, 2019 - 12:35 EDT

Resolved
This incident has been resolved.
Posted Jun 04, 2019 - 20:46 EDT
Monitoring
This issue has been mitigated and service has been restored. We will continue to monitor for problems.
Posted Jun 04, 2019 - 17:15 EDT
Investigating
We are investigating reports of system slowness in the Customer Portal and Platform API. We will update as soon as we have more information.
Posted Jun 04, 2019 - 17:02 EDT
This incident affected: Customer Portal and Platform API.