On Wednesday September 28th, 2022 starting at approximately 1:35PM ET, we observed an increase in the number of 500 errors experienced in our Customer Portal. This affected all accounts with Listings features enabled, in both sandbox and production environments. Production recovered at 1:53PM ET and sandbox at 1:57PM ET.
A change deployed to a downstream service resulted in an error rendering the UI of our Customer Portal. Reverting this change resolved the problem.
We will be expanding the scope of our acceptance testing to allow us to detect issues that may only affect a subset of accounts and prevent them being deployed to sandbox or production. We also plan to harden the core of our Customer Portal so it can degrade more gracefully in light of errors in downstream services.