Pages Locators Errors
Incident Report for Yext
Postmortem

Summary

On Tuesday, February 14, 2023, a change was deployed at 11:27AM ET which caused Locators for a subset of Classic Pages sites to return no results. The Yext engineering team was alerted to the issue at 12:10PM ET and a bug was identified and rolled back at 12:26PM ET, causing Locators to resume normal functionality.

Root Cause

The engineering team was removing deprecated monitoring code from their services. One of the changes to remove this functionality incorrectly removed a section of code used to propagate Live API responses to the Locator frontend. 

Remediation

The issue was resolved when the bug was identified and rolled back. The bug was subsequently fixed and the updated code was later deployed. We plan on adding more thorough testing to the deployment pipeline for the Locator serving system to ensure responses contain the expected results, as well as add monitoring for empty results in Locator responses. The engineering team will also be scrutinizing their deployment and manual testing processes to ensure bugs like this are caught before they reach production in the future.

Posted Feb 22, 2023 - 14:01 EST

Resolved
This incident has been resolved.
Posted Feb 14, 2023 - 15:25 EST
Monitoring
We have remediated the errors. We will continue to monitor the situation.
Posted Feb 14, 2023 - 12:52 EST
Identified
We have identified the issue and are working on remediation.
Posted Feb 14, 2023 - 12:40 EST
Update
We are continuing to investigate this issue.
Posted Feb 14, 2023 - 12:38 EST
Investigating
We are investigating reports of errors in for a subset of customers that are using Pages Locators
Posted Feb 14, 2023 - 12:21 EST
This incident affected: Content (Content API), Pages (Pages Serving), and Search (Search Serving).