Starting on May 11th 2021 at approximately 8.00 p.m ET, the majority of Pages Generation tasks stalled, delaying updates and publishes of sites. A fix was applied on May 12th 2021 at 12.47 a.m. ET and full service was restored by 1.30 a.m. ET. At this point, all sites were regenerated in full to ensure they were up to date. No data was lost and serving was unaffected.
An error in a regular cleanup task resulted in some generation tasks being unable to complete. This prevented new site generation tasks from starting. The cleanup task was disabled to permit a reset that unblocked generations.
The cleanup task mentioned in the root cause will be updated to prevent the situation that caused this incident from occurring again, and we will be improving our monitoring to alert the Engineering team sooner should any publishes stall out in this way in future.