On September 7th at 12:23 p.m. ET, the Customer Portal experienced increased latency and occasional timeouts. By 1:33 p.m. ET, the cause was identified, and the Customer Portal returned to normal.
New software was deployed that introduced significant overhead to pages within the Customer Portal.
The fix was to roll back the new code. We will re-implement the feature in a less resource-intensive manner.
The problem was not noticed in pre-production environments because it only manifested under production-level workloads. This class of problem will be handled going forward by better telemetry and alerting regarding abrupt changes in Customer Portal resource utilization.