On November 16, 2021, at 10:27AM ET, we received reports of errors when creating social posts on the Customer Portal. At 10:35AM, the root cause was identified as a long-running database cleanup operation that locked a database used during social post creation. The cleanup operation was stopped at 10:38AM ET and social posting errors were no longer observed in the Customer Portal. No data loss of social posts created in the Customer Portal occurred during this timeframe.
The root cause of the incident was a long-running cleanup operation of deprecated social posting data, which held a lock on a database used in social post creation workflows. This prevented new social posts from being created via the Customer Portal or via certain v1 endpoints of the Platform API, until that lock was released.
The issue was remediated by terminating the long-running cleanup operation and replacing it with batches of smaller operations that would not lock the database. This batching approach will be used in future cleanup operations. Additionally, we will be restructuring the affected database relationships to remove key constraints that would cause similar lock contention issues.