Pages 404 Errors
Incident Report for Yext
Postmortem

Summary

On August 12, starting at 2:33 PM, updates to a subset of sites hosted by Yext Pages resulted in many web pages on those sites being removed rather than updated. At 3:59 PM, we corrected the underlying issue and began restoring the affected sites. By 4:59 PM, all pages had been restored, though a small portion were not updated to reflect data changes made on August 12 until 7:05 PM.

Consumers requesting a page that had been removed saw a "not found" error page instead, or were redirected to a search or directory page on the same site. The issue was limited to pages for individual locations or entities; store locators were not affected.

Root Cause

A change to the system that updates Yext Pages sites contained a bug affecting only certain types of sites that automated and manual testing failed to catch before production deployment.

Remediation

We are adding additional automated tests to prevent a recurrence of this class of bug. Additionally, we are expanding our active monitoring to be able to detect this type of issue more quickly, and improving the speed of our processes to restore service in an emergency using recent backups.

Posted Aug 16, 2019 - 07:31 EDT

Resolved
All pages have been returned to showing the correct up-to-date content. This issue is now resolved.
Posted Aug 12, 2019 - 19:05 EDT
Monitoring
We have implemented mitigations that should have eliminated unexpected 404 errors, though some pages may be exhibiting stale content. We are continuing to work toward full resolution.
Posted Aug 12, 2019 - 17:10 EDT
Identified
We have identified the issue and are implementing mitigations.
Posted Aug 12, 2019 - 16:37 EDT
Investigating
We are investigating reports of some location pages returning 404 errors.
Posted Aug 12, 2019 - 16:25 EDT
This incident affected: Pages Serving.