Subscribe to our newsletter
Get the latest news and industry insights in your inbox.
Subscribe to our newsletter
Thanks for subscribing
Altitude 2015, our first-ever customer summit, provided a great opportunity to hear from our customers about how they use Fastly. In this post, I’ll share an in-depth overview of Joe Williams’ talk on mitigating security threats with a CDN — full video and slides are below.
Joe is a Computer Operator at GitHub, the web-based Git repository hosting service used by more than 10 million people. Joe mostly works on the load-balancing tier, and he and his team have developed techniques and built tools to deal with the various attack vectors they’ve seen. In his talk, he emphasizes that you can deploy these best practices on your own sites, in the event of a distributed denial of service (DDoS) attack.
There are number of common techniques that GitHub uses during DDoS attacks:
Joe noted that one of the things GitHub does well when mitigating an attack is fast edge configuration changes.
GitHub has “a pretty healthy graphing culture.” They keep track of all their traffic and their providers, enabling them to monitor site health in real time. They monitor how many requests are getting “hit by the banhammer” (banning a client) — by tracking how many requests are sent to certain backends (set via HAProxy), they can look for anomalies. They keep a lot of their provider graphs in Graphite dashboards — including Fastly’s stats API.
Another part of visibility is logging different types of headers; Joe notes that “HAProxy makes this easy,” allowing them to do so regularly.
GitHub does a lot of configuration testing for their edge devices and services as well as for their providers — they have an extensive testing suite for Fastly. They test every single access control list (ACL) in the HAProxy configuration; though it took six months to set up, Joe emphasized that it was “well worth it,” as it ensures changes have the effect the want when they’re rolled out.
In the future, GitHub is going to need “better tools that identify human traffic.” Joe is looking into various traffic authorization tools, such as captchas, that they can turn on during an attack to determine whether a request is coming from a human.
One of Joe’s current projects is switching away from monolithic load balancing to a service-based load balancing tier; he wants to set up load balancing for each individual service supporting github.com, including pages and api.github.com. This will allow them to spread load balancing across more machines, giving attacks a smaller blast radius so GitHub can isolate attack traffic to a specific service.
Joe also plans on switching to more granular per-service maps (they’re currently on one map). Fastly has the ability to push GitHub’s traffic to POPs that aren’t already flooded, giving Joe’s team another tool for mitigating attacks.
Check out Joe’s talk below, and read the full summary of Altitude here. Stay tuned as we recap more Altitude 2015 talks.
Securing online transactions: announcing our plan for TLS 1.0 and 1.1 deprecation
The PCI DSS 3.1 standard has changed. In order to keep you up-to-date and secure online, we’re announcing our plan for TLS 1.0 and 1.1 deprecation.
Engineering a more resilient internet
Fastly Director of Security Engineering Maarten Van Horenbeeck shares his experiences of how the security community can protect the “global commons” that the internet has become.
How to fuzz a server with American Fuzzy Lop
In this blog post, I’ll describe how to use AFL’s experimental persistent mode to blow the doors off of a server without having to make major modifications to the server’s codebase. I’ve used this technique…