Delay in adding/updating entities in Answers
Incident Report for Yext
Postmortem

Summary

Beginning at 9.30am ET on Tuesday September 09th 2020, a reindexing operation resulted in a large backlog in the update queue for Answers. As such, content for new entities and updates to existing entities were delayed in appearing in Answers results. This backlog was cleared at 12.30pm ET.

This issue recurred at 9.30am ET  the following day (September 10th 2020). A fix for the underlying issue was deployed shortly after this, preventing future recurrences. The resulting backlog was processed by 11.30am.

Root Cause

A change in commonly used subscription data caused a storm of event notifications which overloaded the Answers backend. This resulted in a large number of businesses being reindexed, which created the large backlog.

Remediation

We are adjusting the processing of Answers updates to be more resilient to large bursts of updates such as this.

We are also improving our monitoring of these queues to ensure we are proactively notified and react more quickly if any event causes Answers updates to have delays.

Posted Sep 19, 2020 - 16:14 EDT

Resolved
This incident has been resolved.
Posted Sep 15, 2020 - 13:19 EDT
Monitoring
The update backlog has been fully processed and we will continue to monitor the situation.
Posted Sep 15, 2020 - 12:33 EDT
Identified
We've identified the source of the delay and are adding capacity to process the backlog.
Posted Sep 15, 2020 - 12:10 EDT
Investigating
We are currently investigating a delay in processing of updates to Entities in Answers. This is resulting in changes to Entities over the last few hours not being reflected in Answers results.
Posted Sep 15, 2020 - 12:08 EDT
This incident affected: Search (Search Serving).