Pages Publishing Degradation
Incident Report for Yext
Postmortem

Summary

On January 29th, 2021, beginning at 6:17 p.m. ET, the Pages publishing infrastructure encountered a bug which caused a single site to be published incorrectly. Yext engineers restored the affected site at 7:47 p.m. ET to its previous version. Pages publishing was paused at 7:49 p.m. ET to prevent any further erroneous data updates and to perform root cause analysis and testing. Once the issue had been identified and isolated, the publishing flow was re-enabled at 8:38 p.m. ET.

Root Cause

A transient infrastructure timeout at a specific point in the publishing process for the affected site put future publishes for this site in a bad state. A subsequent incremental publish from a data update resulted in the new generated page using references to old versions of assets.

Remediation

We are immediately patching the publishing system to protect against this transient error. Additionally, we will be building a failsafe system to prevent any updates that reference out-of-date configuration.

Posted Feb 05, 2021 - 18:20 EST

Resolved
This incident has been resolved.
Posted Jan 28, 2021 - 23:07 EST
Monitoring
We have re-enabled publishing functionality after performing extensive testing to verify the correctness of publishes. We will continue to monitor for any regressions.
Posted Jan 28, 2021 - 20:56 EST
Investigating
We are currently investigating reports of missing assets in a small number of Pages. We have temporarily disabled publishing to avoid any additional data errors, and we currently believe there is no impact to Pages Serving. We will update as soon as we have more information.
Posted Jan 28, 2021 - 20:07 EST
This incident affected: Pages (Pages Serving, Pages Generation).