Page Builder sites missing content
Incident Report for Yext
Postmortem

Summary

On November 27 and 28, 2019, most pages created using the Yext Page Builder displayed incorrectly, with generic placeholder images and missing entity-specific content.

The issue affected any pages that were updated starting around 10:00 a.m. US Eastern Time on November 27. At around 4:00 a.m. on November 28, a nightly process that regenerates all Yext Page Builder pages resulted in the issue expanding to all pages for individual entities.

At 11:10 a.m., Yext engineers were able to revert all sites to a backup taken around 2:30 a.m. on November 28, reducing the scope of the issue back to those pages updated between 10:00 a.m. on November 27 and 2:30 a.m. on November 28. Recovery efforts continued, with all pages restored to their proper contentful state by 2:28 p.m.

Root Cause

A change to how the Page Builder system incorporates data from the Yext Knowledge Graph was incorrectly rolled out. Although all Page Builder sites were updated to use the new method upon their next template update, the configurations used for immediate publishes were inadvertently left untouched. When support for the previous method was removed, sites whose templates had not been updated since November 18 could no longer load Knowledge Graph data.

Remediation

We will automate the process of rolling out similar configuration changes, to reduce the probability of similar missed steps from happening in future updates. We will also be adding extra safeguards to flag similar situations as errors to decrease the time to discovery of future similar issues.

Posted Dec 12, 2019 - 12:31 EST

Resolved
We have completed our monitoring period, all services continuing to function normally. This issue is resolved.
Posted Nov 28, 2019 - 16:37 EST
Monitoring
We have restored all sites back to their fully up-to-date content. Updates are also now publishing as normal.

We will continue monitoring for any regressions.
Posted Nov 28, 2019 - 14:31 EST
Update
We have been able to restore most sites hosted by Yext Pages to their fully up-to-date content, including all sites managed by Yext Consulting. A limited number of sites built with the Yext Page Builder are still showing their content from overnight, and may still have a few pages missing content. We are continuing to work on restoring all sites as soon as possible.
Posted Nov 28, 2019 - 13:57 EST
Identified
We have identified why pages created using the Yext Page Builder were missing content. We are currently working on regenerating these pages to restore normal service.

Although most pages on sites created with the Yext Page Builder are currently showing the correct content (as of around 2:30 a.m. US Eastern Time), we have found that some pages on those sites that were updated yesterday (US time) are still missing their content. We are looking into how to restore these pages as quickly as possible.

All pages on sites managed by Yext Consulting should be displaying as normal, except for missing any updates made after around 2:30 a.m. US Eastern Time.
Posted Nov 28, 2019 - 13:02 EST
Update
We are continuing to investigate the root cause of the missing content. We are continuing to hold all Yext Pages sites, including both sites created using the Yext Page Builder and sites managed by Yext Consulting, at their content as of around 2:30 a.m. US Eastern Time.
Posted Nov 28, 2019 - 12:31 EST
Update
We have mitigated the problem by resetting the data on all sites hosted by the Yext Pages platform to their state from early this morning (New York time). Updates to site content are currently paused as we continue to investigate the root cause of the missing content.
Posted Nov 28, 2019 - 11:17 EST
Investigating
We are investigating why many pages on sites created using the Yext Page Builder are missing their content.
Posted Nov 28, 2019 - 11:01 EST
This incident affected: Pages Serving.