From 3:00 p.m. US Eastern Time on November 28 until 5:47 p.m. on December 1, the Yext Pages platform intermittently displayed stale content for some web pages. When these pages were affected, they displayed content from the early morning of November 28 (US Eastern Time), not reflecting any updates that had been made after that point. The issue mostly affected the sites of our Asia-Pacific customers, which generally showed the November 28 content throughout this time period. Other customers were more likely to see a small minority of individual pages on their sites randomly affected for two hours at a time.
During the mitigation of a separate issue, the Yext Pages platform was set to serve from its daily backup from November 28. During that incident, some ad hoc configuration changes were made to restore regular updating to most sites more quickly than supported by the usual interfaces for configuring the backup serving system. At the end of the incident, as the ad hoc changes were reversed to return to normal service, our Asia-Pacific data center was mistakenly left still configured to serve the November 28 backups.
Though the issue primarily affected consumers in the Asia-Pacific region, the global distribution of cached content by our CDN also caused intermittent issues in the rest of the world.
We will upgrade the interfaces for interacting with the backup serving configuration system to reduce the need for error-prone ad hoc operations. We will also add automated monitoring and alerting to ensure forgotten settings are discovered much more quickly.