Pages sites displaying stale content
Incident Report for Yext
Postmortem

Summary

From 3:00 p.m. US Eastern Time on November 28 until 5:47 p.m. on December 1, the Yext Pages platform intermittently displayed stale content for some web pages. When these pages were affected, they displayed content from the early morning of November 28 (US Eastern Time), not reflecting any updates that had been made after that point. The issue mostly affected the sites of our Asia-Pacific customers, which generally showed the November 28 content throughout this time period. Other customers were more likely to see a small minority of individual pages on their sites randomly affected for two hours at a time.

Root Cause

During the mitigation of a separate issue, the Yext Pages platform was set to serve from its daily backup from November 28. During that incident, some ad hoc configuration changes were made to restore regular updating to most sites more quickly than supported by the usual interfaces for configuring the backup serving system. At the end of the incident, as the ad hoc changes were reversed to return to normal service, our Asia-Pacific data center was mistakenly left still configured to serve the November 28 backups.

Though the issue primarily affected consumers in the Asia-Pacific region, the global distribution of cached content by our CDN also caused intermittent issues in the rest of the world.

Remediation

We will upgrade the interfaces for interacting with the backup serving configuration system to reduce the need for error-prone ad hoc operations. We will also add automated monitoring and alerting to ensure forgotten settings are discovered much more quickly.

Posted Dec 12, 2019 - 12:33 EST

Resolved
From November 28 through December 1, content from the early morning of November 28 was sometimes displayed for some pages hosted by the Yext Pages platform. The issue that caused this stale content to be displayed sometimes was resolved at around 5:45 p.m. on December 1.
Posted Dec 01, 2019 - 18:00 EST