Summary

On Thursday, August 9, from approximately 5:10 AM to 6:57 AM EDT, Analytics data was unavailable from the Customer Portal and the API. No data was lost and no incorrect data was returned.

Root Cause

A previously-unseen failure mode in a third party provider caused our data analytics system to temporarily refuse all requests. This event also alerted our team, and manual intervention quickly resolved the situation. In response to this issue, we have made our system robust to this and related failure modes.

Posted Aug 15, 2018 - 21:59 EDT

Resolved

The incident has been fully resolved and we are working to design and implement mitigation steps to prevent future occurrences.

Posted Aug 09, 2018 - 11:25 EDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Aug 09, 2018 - 07:12 EDT

Update

We have identified the issue and restored full service. We are continuing to monitor the situation and look for other related issues.

Posted Aug 09, 2018 - 07:11 EDT

Investigating

Monitoring has alerted us that requests for analytics data are timing out or failing in the customer portal. We are investigating the cause and will update this when we learn more.

Posted Aug 09, 2018 - 07:02 EDT

This incident affected: Customer Portal Login.