Delays Populating Answers Analytics
Incident Report for Yext
Postmortem

Summary

Around 1PM ET on February 4th 2021, telemetry indicated that Answers query logs were not being populated into our Analytics system, thus preventing them from appearing in several places within the Answers and Analytics platform.  Engineers identified the problem and were able to backfill the queries and repair the pipeline into Analytics by 4:45PM ET.  No data was lost.

Root Cause

The piece of Yext software which processes Answers query logs was unable to properly interface with the open source software which stores the queries.  The problem was relatively simple, but was exacerbated because we did not have an easy flag to enable the required debugging.

Remediation

We have designed tooling to facilitate debugging in real-time, thus enabling us to diagnose issues of this kind more readily.

Posted Feb 16, 2021 - 16:22 EST

Resolved
This incident has been resolved.
Posted Feb 04, 2021 - 18:54 EST
Monitoring
A fix has been implemented, and Answers Searches are now populating into the various places that utilize them. It will take some time before the pipelines have fully caught up. We will monitor the system through this process.
Posted Feb 04, 2021 - 16:47 EST
Identified
We have identified the cause and are working on a solution.
Posted Feb 04, 2021 - 16:19 EST
Update
We are continuing to investigate this issue.
Posted Feb 04, 2021 - 14:48 EST
Investigating
We are investigating delays getting recent Answers searches to appear in Answers Analytics storage. This may affect various views including Answers Analytics, Answers Overview and Answers Search Terms. Answers Serving is unaffected.
Posted Feb 04, 2021 - 13:18 EST
This incident affected: Customer Portal Login.