Starting on May 5th 2021 at 10.26 a.m. ET, the Customer Portal began serving errors to all users. The issue was corrected by 10.40 a.m. ET, and the number of errors being served began to decline, with full service resumed by 11.03 a.m. ET. No data was lost.
A configuration change was applied to our production network resulting in a database server being disconnected from the network. This change was the result of a typo that was not caught by a code reviewer. The change was reverted as soon as our monitoring reported the resulting failures.
We will be updating our procedures to make errors of this class more obvious.