Increased Error Rates in Customer Portal
Incident Report for Yext
Postmortem

Summary

Starting at 4:02 PM ET on October 7, 2018, a small portion of the pages in our customer portal were unavailable. Service was incrementally restored, and fully available by 9:14 PM ET.

Consumer facing services were unaffected. Underlying data was unaffected.

Root Cause

During routine patching of systems, a portion of our compute nodes experienced failures in their network interfaces. These issues were not seen when applying the same patches to our pre-production environments. We were able to move the workloads on these nodes to other nodes to mitigate the issue and restore the full environment shortly after.

We are adjusting our patching process to make some of our pre-production environments more closely match production configurations and load.

Posted about 2 months ago. Oct 11, 2018 - 10:01 EDT

Resolved
All services have been restored and monitoring indicates no remaining issues.
Posted 2 months ago. Oct 08, 2018 - 12:26 EDT
Monitoring
The remaining issues have been mitigated. We will continue to monitor for any issues.
Posted 2 months ago. Oct 07, 2018 - 23:57 EDT
Update
We are continuing to work on mitigations for remaining issues.
Posted 2 months ago. Oct 07, 2018 - 19:51 EDT
Update
Most functionality has been restored, and we are continuing to work on mitigations for the remaining issues.
Posted 2 months ago. Oct 07, 2018 - 18:50 EDT
Identified
We have identified the underlying issue and are working on mitigation.
Posted 2 months ago. Oct 07, 2018 - 17:33 EDT
Investigating
We are investigating reports of increased error rates in the customer portal. We will update as soon as we have more information.
Posted 2 months ago. Oct 07, 2018 - 16:54 EDT
This incident affected: Customer Portal and Platform API.