On October 29th, 2024, at approximately 3:00PM ET, the Customer Portal experienced an outage for nearly two hours. Some remaining errors remained for the subsequent 2.5 hours, and the portal was returned to an error-free state by approximately 7:45PM ET. At that point, some translations were unavailable. By 10:00PM ET, the translations were fully restored.
Our front-end web services depend on language translation files to serve the Customer Portal in various languages. These files are hosted on a legacy storage system, and that storage system had become desynchronized with the application servers that utilize them. This caused slowness when accessing these files. The slowness caused a cascade of issues that ultimately resulted in the Customer Portal being unable to start.
To remediate this issue, the entire fleet of application servers had to be restarted in a rolling fashion. This was time-consuming, leading to the protracted downtime.
Prior to this incident, we had already begun deprecating the legacy storage system and had migrated many workloads off it. We will redouble our efforts and prioritize migrating these translation files to the new architecture.
Additionally, we have added telemetry to notify us more quickly if the legacy storage system experiences similar problems in the future.
Lastly, we have refined the process for restarting our fleet of application servers so that we can recover more quickly in the worst-case scenario.