Pages 2.0 publishes and updates delayed

Incident Report for Yext

Postmortem

Summary

On Monday, February 27th, 2023 at 6:00AM ET, Pages 2.0 started having delays for approximately 5% of the pages published and updated. At 6:11AM ET, an alert was sent out from the system about the issue to the engineering team. At 11:11AM ET, a Yext employee notified the team of the problem and the investigation began. At 1:02PM ET the delays were resolved and service levels returned to normal.

Root Cause

A configuration mismatch caused the system to fail when processing large documents.

Remediation

The problem was resolved by deploying an updated configuration file to the system and restarting affected services. The engineering team will be improving alerting to make sure that delays are caught more quickly.

Posted Mar 01, 2023 - 17:27 EST

Resolved

This incident is resolved.
Posted Feb 27, 2023 - 14:54 EST

Monitoring

We have remediated the issue, and are processing the delayed data. We will continue to monitor the situation.
Posted Feb 27, 2023 - 13:10 EST

Identified

We have identified the issue and are working on remediation.
Posted Feb 27, 2023 - 12:57 EST

Update

We are continuing to investigate this issue.
Posted Feb 27, 2023 - 11:42 EST

Investigating

We are investigating delays in publishes and updates for sites running on Pages 2.0.
Posted Feb 27, 2023 - 11:29 EST
This incident affected: Pages (Pages Serving, Pages Generation).