We have completed the investigation into incident 2023-07-10 and the report is available here.
Posted Jul 19, 2023 - 11:02 EDT
Resolved
The platform is fully operational and you may resume normal operations. We will continue to monitor the system for potential performance degradation and remediate issues. We appreciate your patience throughout the service disruption.
Posted Jul 15, 2023 - 11:08 EDT
Monitoring
All systems are fully functional. We will continue to monitor the situation.
Posted Jul 15, 2023 - 10:41 EDT
Update
The ability to filter reviews by language is now available for all reviews.
Posted Jul 15, 2023 - 09:19 EDT
Update
Sandbox environments are open and fully operational.
We’ve restored complete functionality of Reviews, with slight degradation in the ability to filter by language.
Posted Jul 14, 2023 - 16:53 EDT
Update
The Customer Portal and Management APIs have been restored to full service and all heath checks are passing. We will update on other areas shortly.
Posted Jul 14, 2023 - 15:33 EDT
Update
Services for the Customer Portal and Platform APIs are recovering, although some features may still experience errors.
Posted Jul 14, 2023 - 15:24 EDT
Update
We are investigating an outage of the Customer Portal and Management UI. Service will be restored shortly.
Posted Jul 14, 2023 - 15:16 EDT
Update
We continue to work to restore Reviews to fully operational. New reviews are being collected and indexed in real-time. Existing reviews are reindexed, and we have begun work to enable full text search and sentiment analysis. ETA for all reviews data to be available has been updated to July 14th at 9:00am EDT.
We continue to work to restore the Sandbox environment. Sandbox is still on track to be re-opened within the next 12 hours.
Posted Jul 14, 2023 - 05:48 EDT
Update
We continue to work to restore Reviews to fully operational. New reviews are being collected and indexed in real-time. Existing reviews are reindexed, and we have begun work to enable full text search and sentiment analysis. ETA for all reviews data to be available is July 14th at 6:00am EDT.
Managed Service ETLs are now verified and operational.
We continue to work to restore the Sandbox environment. Sandbox is still on track to be re-opened within the next 24 hours.
Posted Jul 13, 2023 - 23:48 EDT
Update
We continue to work to restore Reviews to fully operational. New reviews are being collected and indexed in real-time. Existing reviews are reindexed, and we have begun work to enable full text search and sentiment analysis.
ETLs are enabled and running. Managed Service ETLs now are operating with slight degradation – improved from full disruption. Managed Services will continue to monitor and validate the data.
All Pages sites are all operational.
Sandbox environments are still on track to be re-opened within the next 33 hours.
Posted Jul 13, 2023 - 13:27 EDT
Update
We continue to work to restore Reviews to fully operational. New reviews are being collected and indexed in real-time. Existing reviews are still being reindexed.
ETLs are enabled and running. Managed Service ETLs now are operating with slight degradation – improved from full disruption. Managed Services will continue to monitor and validate the data.
Pages sites are operational. In edge cases, Pages that use review aggregations may not be receiving real-time updates, but are still serving. Landing pages are operating with slight degradation.
Sandbox environments will be re-opened within the next 36 hours.
Posted Jul 13, 2023 - 10:56 EDT
Update
Publisher network is operational.
Posted Jul 12, 2023 - 20:32 EDT
Update
Knowledge Assistant and App Directory are operational.
Posted Jul 12, 2023 - 16:44 EDT
Update
We’ve completed system validation and re-enabled accounts with some capabilities temporarily paused. We’ve taken the step to limit access to reduce the risk of additional impact to your services. Capabilities currently enabled on accounts:
- Login - Update Content - Push Updates to Listings - Push Updates to Search - Push Updates to Live API - Pages Updates will come online as validation continues - Review data will come online as it re-indexes
We will continue to monitor our system and re-open features and capabilities as it returns to being fully operational.
We appreciate your patience throughout this service disruption and will post an incident report once we complete the investigation. Continue to monitor the Yext Trust Site for the latest updates.
Posted Jul 12, 2023 - 16:36 EDT
Update
We continue to work toward opening customer access which we anticipate to enable by later today, July 12th (EDT).
Reindexing has completed for a portion of data thus far and is continuing through today. This is one of the final requirements to re-enable access to the platform.
Consumer-facing experiences continue to be powered. Access to the platform to update data remains closed at this time.
Posted Jul 12, 2023 - 13:04 EDT
Update
We continue to work toward opening customer access, which we anticipate to enable by later today, July 12th (EDT).
Reindexing has completed for a portion of data thus far and is continuing through today. This is one of the final requirements to re-enable access to the platform.
Consumer-facing experiences continue to be powered. Access to the platform to update data remains closed at this time.
Posted Jul 12, 2023 - 12:03 EDT
Update
We continue to work toward opening customer access which we anticipate to enable by later today, July 12th (EDT).
Reindexing has completed for a portion of data thus far and is continuing through today. This is one of the final requirements to re-enable access to the platform.
Consumer-facing experiences continue to be powered. Access to the platform to update data remains closed at this time.
Posted Jul 12, 2023 - 11:04 EDT
Update
We continue to work toward opening customer access which we anticipate to enable by this afternoon, July 12th (EDT).
Reindexing has completed for a portion of data thus far and is continuing through today. This is one of the final requirements to re-enable access to the platform.
Consumer-facing experiences continue to be powered. Access to the platform to update data remains closed at this time.
Posted Jul 12, 2023 - 10:17 EDT
Update
Updating Hitchhikers Site to Partial Outage.
Posted Jul 11, 2023 - 22:18 EDT
Update
We have successfully deployed and tested the majority of services in read/write mode in the backup data center. We are currently executing content reindexing, which is one of the final requirements to re-enable customer access. Based on observed indexing speed we anticipate restoring access tomorrow (July 12) afternoon EDT.
Posted Jul 11, 2023 - 21:37 EDT
Update
Our teams are working to resolve the disruption as quickly as possible. We are currently validating the system before restoring additional services or access.
Posted Jul 11, 2023 - 14:56 EDT
Update
We continue to bring platform services back online. As of this morning, all consumer-serving is restored. We are taking steps to mitigate further disruption by moving the service to the secondary Source Data Region data center, and that work is well underway.
Posted Jul 11, 2023 - 10:54 EDT
Update
We are waiting for an update from our primary Source Data Region data center and will post here when we receive an update.
Last night we failed over from our primary Source Data Region data center to our backup Source Data Region data center.
The system there is in read-only mode, which means that the Platform API is partially available for reads of data available in the platform. However, users will not be able to login to the Customer Portal as logins currently require write access to be enabled. This also means that Yext Employees cannot login to the Customer Portal to make updates on customers' behalf and that updates to Pages, Listings, Search and Reviews are not currently available.
We will provide another update in the next hour.
Posted Jul 11, 2023 - 08:27 EDT
Update
We have failed over critical services to our back up data center and reestablished consumer serving from our global regions in Japan, Germany, and the US. We are leaving the knowledge graph in read-only mode while we wait for our primary data center to come back online around mid-day Tuesday, based on the recovery timeline we received from our contacts at the data center.
Posted Jul 11, 2023 - 02:10 EDT
Update
Service on Platform API (Knowledge and Administrative) is partially restored in read-only mode.
Posted Jul 11, 2023 - 01:00 EDT
Update
We have received another update from our primary Source Data Region data center. Its full text is below:
After reviewing the site, the fire marshall is requiring that we extensively clean the UPS devices and rooms before they will allow us to re-energize the site. We have a vendor at the site currently who will be performing that cleanup. We will provide an update at 8:00AM EDT unless something significant changes overnight.
Posted Jul 11, 2023 - 00:36 EDT
Update
Due to the current outage, Connectors are unavailable, and client ETLs are experiencing delays.
Posted Jul 11, 2023 - 00:02 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 23:22 EDT
Update
We have begun a rolling restart of our consumer-serving jobs to restore service and will continue to monitor activity.
Pages Generation remains unavailable.
Posted Jul 10, 2023 - 23:06 EDT
Update
We are attempting a partial fail over to our backup Source Data Region data center.
The loss of power at our primary Source Data Region data center disconnected our Consumer Serving Regions from our primary code repository system as well as our secure secrets vault.
We are working to deploy a mitigation to allow us to deploy from our backup Source Data Region data center.
Posted Jul 10, 2023 - 21:55 EDT
Identified
We have received an update from our primary Source Data Region data center. Its full text is below:
Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.
We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process.
We are currently waiting on the fire marshal and local utility to re-energize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour. At the current time, the fire department is controlling access to the building and we will not be able to let customers in.
Posted Jul 10, 2023 - 21:53 EDT
Update
We are continuing to investigate this issue.
Posted Jul 10, 2023 - 21:52 EDT
Update
We are actively working to restore access to our consumer-serving services and waiting for further updates from our primary data center.
Posted Jul 10, 2023 - 20:33 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 19:49 EDT
Update
Our primary data center is experiencing an outage. We are working to minimize impact.
Posted Jul 10, 2023 - 19:23 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 19:03 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 19:00 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 18:36 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 18:34 EDT
Update
Updating list of impacted services.
Posted Jul 10, 2023 - 18:25 EDT
Update
Live API, Search Serving, Pages Generation, the Hitchhiker Site, Analytics & Reviews are also impacted.
Posted Jul 10, 2023 - 18:17 EDT
Investigating
We are investigating an increased rate of errors across the Yext platform.