Summary

Between approximately 12:00PM ET Thursday January 25th, 2024 and 10:00AM ET on Wednesday January 31st 2024, requests to view analytics data in our customer portal experienced increased latencies, sometimes resulting in timeout errors. This primarily affected customers in our EU environment.

Root Cause

We applied enhancements to our analytics database on Jan 24th after testing these changes in pre-production environments. This resulted in a build up of fragmented data that manifested as performance issues a few days later. The EU environment was more heavily affected due to the lower rate of data being processed.

Remediation

Working with our database vendor, we applied changes in configuration to resolve the conflict and optimize the fragmented data. We will be reviewing our practices for rolling out migrations of this nature to allow additional observation in pre-production environments.

Posted Feb 06, 2024 - 17:18 EST

Resolved

This incident has been resolved.

Posted Jan 31, 2024 - 17:30 EST

Monitoring

Remediations on key affected tables in the US databases have completed. We will continue to monitor the situation.

Posted Jan 31, 2024 - 09:44 EST

Update

The process of applying remediations in our US databases is continuing and we have not seen evidence of errors in either the EU or the US since they started applying. We will provide a further update in the morning, Eastern Time.

Posted Jan 30, 2024 - 23:08 EST

Update

Remediations have been applied in our EU databases and we are starting to see performance returning to normal. Similar operations are being performed on the US databases and we will post further updates as they progress.

Posted Jan 30, 2024 - 19:44 EST

Update

We are applying additional remediations recommended by our analytics database provider and are monitoring them as they roll out. We will provide a further update within the next three hours.

Posted Jan 30, 2024 - 17:14 EST

Update

Following last night's changes, we continue to see timeouts for analytics data in the EU region. We are currently rebuilding affected database tables with a goal of improving performance. This should not result in any loss of data.

Posted Jan 30, 2024 - 11:27 EST

Update

Based on recommendations from our database provider, we have applied configuration changes that should improve query times as more data comes in. We will leave these running overnight and provide an update tomorrow morning, Eastern time.

Posted Jan 29, 2024 - 23:07 EST

Update

We are continuing to work with our analytics database vendor to resolve the query optimization issue. They have escalated the ticket to their engineering team for further assistance.

Posted Jan 29, 2024 - 17:23 EST

Update

We are working with our analytics database vendor to resolve an issue where some data after Jan 24 is not being optimized for querying. In the meantime, analytics queries with date ranges ending before Jan 24 should perform as normal.

Posted Jan 29, 2024 - 14:13 EST

Identified

The issue has been identified and we are working on a fix.

Given the nature of the root cause, we believe it also may be impacting some queries in the US region.

We will continue to provide updates as we make progress.

Posted Jan 29, 2024 - 11:41 EST

Investigating

We are currently looking into reports of query timeouts leading to errors in our analytics platform. Reports so far have been limited to our EU region, but we will confirm this as part of the investigation.

Posted Jan 29, 2024 - 09:47 EST

This incident affected: Analytics (Analytics Queries).