Between 5:30PM ET on Tuesday, March 28th, 2023, and 6:15PM ET on Wednesday, March 29th, 2023, a subset of data ETLs managed by Yext Services were delayed in completing their execution. The engineering team was notified of potential issues at 11:30AM ET on the morning of March 29th and were able to implement a fix that allowed further ETL invocations to complete processing on their normal schedule.
A connection resource leak in our routing infrastructure starved out microservices that the Managed Services ETLs depended on for processing.
Changes have been made to mitigate the resource leak, as well as improve overall performance through resource caching. Monitoring improvements will be implemented to more quickly identify any similar issues in the future.