On March 5, 2025, at approximately 7:00AM ET, Content API, Search, Knowledge Tags, and Pages started experiencing intermittent failures for a subset of customers. By approximately 9:30AM ET, service was restored to almost all customers, and by 11:00AM it was restored to all customers.
The root cause was determined to be a distributed denial of service (DDoS) attack on a subset of our Content API resources. These attacks caused excessive load on particular shards of our backend data layer, causing requests to those specific shards to experience high latencies and timeouts. This, in turn, caused a partial outage in Content API, as well as other Yext services that use Content API, including Search, Knowledge Tags, and Pages.
While the system already has multiple layers of protection against denial of service attacks (such as rate-limiting and caching), we will implement additional defense mechanisms at both the CDN and backend levels to further neutralize this specific distributed attack vector. Additionally, we will investigate introducing further short-circuiting capabilities at the data storage layer to mitigate general sharp spikes in load.