Increased error rate in Customer Portal
Incident Report for Yext
Postmortem

Summary

On Wednesday September 28th, 2022 starting at approximately 1:35PM ET, we observed an increase in the number of 500 errors experienced in our Customer Portal. This affected all accounts with Listings features enabled, in both sandbox and production environments. Production recovered at 1:53PM ET and sandbox at 1:57PM ET.

Root Cause

A change deployed to a downstream service resulted in an error rendering the UI of our Customer Portal. Reverting this change resolved the problem.

Remediation

We will be expanding the scope of our acceptance testing to allow us to detect issues that may only affect a subset of accounts and prevent them being deployed to sandbox or production. We also plan to harden the core of our Customer Portal so it can degrade more gracefully in light of errors in downstream services.

Posted Oct 03, 2022 - 10:08 EDT

Resolved
This incident has been resolved.
Posted Sep 28, 2022 - 15:31 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 28, 2022 - 14:00 EDT
Identified
A fix has been deployed to our production environment. The sandbox environment is still experiencing errors.
Posted Sep 28, 2022 - 13:53 EDT
Investigating
We are currently investigating a high level of 500 errors in the Customer Portal.
Posted Sep 28, 2022 - 13:51 EDT
This incident affected: Customer Portal Login.