Delays in Pages Generation
Incident Report for Yext
Postmortem

Summary

Starting on May 11th 2021 at approximately 8.00 p.m ET, the majority of Pages Generation tasks stalled, delaying updates and publishes of sites. A fix was applied on May 12th 2021 at 12.47 a.m. ET and full service was restored by 1.30 a.m. ET. At this point, all sites were regenerated in full to ensure they were up to date. No data was lost and serving was unaffected.

Root Cause

An error in a regular cleanup task resulted in some generation tasks being unable to complete. This prevented new site generation tasks from starting. The cleanup task was disabled to permit a reset that unblocked generations.

Remediation

The cleanup task mentioned in the root cause will be updated to prevent the situation that caused this incident from occurring again, and we will be improving our monitoring to alert the Engineering team sooner should any publishes stall out in this way in future.

Posted May 17, 2021 - 19:13 EDT

Resolved
This incident has been resolved.
Posted May 12, 2021 - 02:11 EDT
Monitoring
A fix has been applied and the backlog of Pages Generations is decreasing, we will continue to monitor the situation.
Posted May 12, 2021 - 00:47 EDT
Investigating
We are currently investigating reports of delays in starting Pages Generation jobs.
Posted May 12, 2021 - 00:20 EDT
This incident affected: Pages (Pages Generation).