Summary

Beginning on Sunday, March 3rd, 2019, at 11:18 AM EST, engineering was alerted of elevated error rates in the Platform API. Mitigation efforts began immediately, and a fix was deployed by 11:42 AM, at which point error rates returned to normal.

Root Cause

An routine upgrade which had increased the serving capacity for an API endpoint caused errors under extremely high load. Reverting the change reduced errors immediately. Going forward, we plan to double the server capacity and add additional load based testing to catch these issues.

Posted Mar 08, 2019 - 11:48 EST

Resolved

This incident has been resolved.

Posted Mar 03, 2019 - 17:25 EST

Monitoring

We have identified and mitigated the cause of errors and service has been restored. We will continue to actively monitor for errors.

Posted Mar 03, 2019 - 11:42 EST

Investigating

We are investigating reports of elevated error rates in the Platform API. We will post updates as soon as we have more information.

Posted Mar 03, 2019 - 11:18 EST

This incident affected: Content (Management API).