October 24th, 2019, beginning at around 4:00 a.m. Eastern, Yext engineers began observing high error rates in the Analytics service in Sandbox. Service was restored at 12:41 p.m. Eastern. During this outage, Analytics was unavailable in Sandbox. There was no impact to our production services.
During a routine cluster maintenance, a configuration change failed to be deployed to all networking layers, preventing requests from reaching the backend cluster.
We will improve our deployment procedures in our sandbox infrastructure to avoid this class of problem in the future.