On November 27 and 28, 2019, most pages created using the Yext Page Builder displayed incorrectly, with generic placeholder images and missing entity-specific content.
The issue affected any pages that were updated starting around 10:00 a.m. US Eastern Time on November 27. At around 4:00 a.m. on November 28, a nightly process that regenerates all Yext Page Builder pages resulted in the issue expanding to all pages for individual entities.
At 11:10 a.m., Yext engineers were able to revert all sites to a backup taken around 2:30 a.m. on November 28, reducing the scope of the issue back to those pages updated between 10:00 a.m. on November 27 and 2:30 a.m. on November 28. Recovery efforts continued, with all pages restored to their proper contentful state by 2:28 p.m.
A change to how the Page Builder system incorporates data from the Yext Knowledge Graph was incorrectly rolled out. Although all Page Builder sites were updated to use the new method upon their next template update, the configurations used for immediate publishes were inadvertently left untouched. When support for the previous method was removed, sites whose templates had not been updated since November 18 could no longer load Knowledge Graph data.
We will automate the process of rolling out similar configuration changes, to reduce the probability of similar missed steps from happening in future updates. We will also be adding extra safeguards to flag similar situations as errors to decrease the time to discovery of future similar issues.