On Tuesday, September 14th, 2021, starting at 5:55 p.m. ET, updates to listings via the Knowledge Network began experiencing extended delays, with no updates being sent until 6:41 p.m. ET. Delays continued for a small number of listings until approximately 11:00 a.m. ET on September 15th, when normal service resumed.
A backend service responsible for queueing updates for listings failed during a period of high load, and recovery operations resulted in some updates being lost. These updates were recovered by re-publishing the listings for any Entities that had been edited during the time of the outage.
The backend service that experienced the outage has had additional resources applied to handle greater load, and we plan to update monitoring and management of our pipeline to better detect and control such load in future.
We will also be updating our procedures for recovering updates to