September 3, 2020
Please see below for a Fastly Security Advisory (FSecA) outlining an investigation into log misrouting caused by a rare series of conditions.
It is our goal in this FSecA to explain that we clearly understand the applicability and impact of this investigation, and describe the remediation that has been implemented.
On July 29th at 00:00 UTC, Fastly was notified by a customer (customer X) that a single log line intended for a different customer (customer Y) was received by customer X’s log system. Fastly promptly began to investigate and determined that when a complex series of conditions occur, a log line may be misrouted to an incorrect logging service. We were able to trace the root cause to an error in logic introduced by Fastly to improve performance in April 2012. This single report from one customer is the only instance that Fastly is aware of, where all necessary conditions aligned simultaneously in eight years.
Fastly has narrowed this incident to occur under the following combination of conditions (all of the below must be true):
We believe we would have been notified previously had these conditions all been triggered with any frequency in the past 8 years. We take any potential data corruption scenario very seriously (see “Incorrect service routing involving HTTP/2 client connections” and “Request body disclosure to other Fastly services”). As in this case, even one errant log line is not something we accept without root cause analysis. We have put protections in place to avoid this issue in the future.
A fleetwide deployment of a solution that ensures Varnish handles an out-of-memory condition during writing a log line (condition #2 above) has been deployed to prevent the combination of these conditions from causing the log data to be misrouted. Since the application of the fix, if an error occurs while constructing a log event, the process will fail and discard the line instead of writing it. The fleetwide remediation was launched as a phased deployment on July 31st at 10:51 UTC and was completed on August 9th at 14:49 UTC.
Fastly conducted an extensive review of the error conditions following remediation deployment from August 13th through September 2nd to determine the potential rate of occurrence that log lines could have been misrouted. As a result of our review, we do not recommend any actions by our customers. If you have received an unexpected log line, please reach out and we will immediately engage to determine if it is related to this issue.
Events of 2020 July 29th
Events of 2020 July 31st
Events of 2020 August 9
Events of 2020 August 13th through 2020 September 2nd