Errors in updates to social media posts in Platform API
Incident Report for Yext
Postmortem

Summary

On November 16, 2021, at 10:27AM ET, we received reports of errors when creating social posts on the Customer Portal. At 10:35AM, the root cause was identified as a long-running database cleanup operation that locked a database used during social post creation. The cleanup operation was stopped at 10:38AM ET and social posting errors were no longer observed in the Customer Portal. No data loss of social posts created in the Customer Portal occurred during this timeframe.

Root Cause

The root cause of the incident was a long-running cleanup operation of deprecated social posting data, which held a lock on a database used in social post creation workflows. This prevented new social posts from being created via the Customer Portal or via certain v1 endpoints of the Platform API, until that lock was released.

Remediation

The issue was remediated by terminating the long-running cleanup operation and replacing it with batches of smaller operations that would not lock the database. This batching approach will be used in future cleanup operations. Additionally, we will be restructuring the affected database relationships to remove key constraints that would cause similar lock contention issues.

Posted Dec 08, 2021 - 14:38 EST

Resolved
This incident is resolved.
Posted Nov 16, 2021 - 11:53 EST
Monitoring
We have remediated the errors. We will continue to monitor the situation.
Posted Nov 16, 2021 - 10:38 EST
Identified
We have identified the source of errors and are working on remediation.
Posted Nov 16, 2021 - 10:35 EST
Investigating
We are investigating reports of errors in updates to social media posts in the Platform API.
Posted Nov 16, 2021 - 10:27 EST
This incident affected: Customer Portal, Platform API (Knowledge and Administrative), and Knowledge Network.