Subscribe to our newsletter
Get the latest news and industry insights in your inbox.
Subscribe to our newsletter
Thanks for subscribing.
Altitude 2015, our first-ever customer summit, provided a great opportunity to hear from our customers about how they use Fastly. In this post, I’ll share an in-depth overview of Joe Williams’ talk on mitigating security threats with a CDN — full video and slides are below.
Joe is a Computer Operator at GitHub, the web-based Git repository hosting service used by more than 10 million people. Joe mostly works on the load-balancing tier, and he and his team have developed techniques and built tools to deal with the various attack vectors they’ve seen. In his talk, he emphasizes that you can deploy these best practices on your own sites, in the event of a distributed denial of service (DDoS) attack.
There are number of common techniques that GitHub uses during DDoS attacks:
Joe noted that one of the things GitHub does well when mitigating an attack is fast edge configuration changes.
GitHub has “a pretty healthy graphing culture.” They keep track of all their traffic and their providers, enabling them to monitor site health in real time. They monitor how many requests are getting “hit by the banhammer” (banning a client) — by tracking how many requests are sent to certain backends (set via HAProxy), they can look for anomalies. They keep a lot of their provider graphs in Graphite dashboards — including Fastly’s stats API.
Another part of visibility is logging different types of headers; Joe notes that “HAProxy makes this easy,” allowing them to do so regularly.
GitHub does a lot of configuration testing for their edge devices and services as well as for their providers — they have an extensive testing suite for Fastly. They test every single access control list (ACL) in the HAProxy configuration; though it took six months to set up, Joe emphasized that it was “well worth it,” as it ensures changes have the effect the want when they’re rolled out.
In the future, GitHub is going to need “better tools that identify human traffic.” Joe is looking into various traffic authorization tools, such as captchas, that they can turn on during an attack to determine whether a request is coming from a human.
One of Joe’s current projects is switching away from monolithic load balancing to a service-based load balancing tier; he wants to set up load balancing for each individual service supporting github.com, including pages and api.github.com. This will allow them to spread load balancing across more machines, giving attacks a smaller blast radius so GitHub can isolate attack traffic to a specific service.
Joe also plans on switching to more granular per-service maps (they’re currently on one map). Fastly has the ability to push GitHub’s traffic to POPs that aren’t already flooded, giving Joe’s team another tool for mitigating attacks.
Check out Joe’s talk below, and read the full summary of Altitude here. Stay tuned as we recap more Altitude 2015 talks.