November 14, 2019
On November 11, 2019, at 21:57 UTC, Fastly deployed a new build of its HTTP/2 termination software to two Fastly cache servers in the Minneapolis-St.Paul (STP) data center. This build contained a processing flaw involving connection re-use between internal Fastly systems (unrelated to HTTP/2 multiplexing), and caused some incoming HTTP/2 requests for Fastly customers’ services to potentially be routed incorrectly to a group of up to 20 different Fastly customers’ services and origins. This led to some client request data being delivered to, and a response returned by, an incorrect customer origin. The customers whose origins erroneously received these requests may have logged the incorrectly-routed request data.
Fastly was first notified by a customer of a client error on November 12, 2019, at 23:07 UTC. On November 13, 2019, at 00:50 UTC, all customer traffic was diverted away from the affected data center. Fastly immediately commenced an investigation, and on November 14, 2019, at 00:31 UTC, we validated the presence of incorrectly routed request data in a customer’s logs.
We estimate this flaw affected 0.00016% of our global request traffic during the 27-hour period. It is unlikely that affected client requests came from outside of North America.
Because Fastly does not store customer log data, we are not able to say with certainty if an affected request was incorrectly routed.