The internet has recently seen what appears to be one of the biggest distributed denial-of-service (DDoS) attacks ever – a sustained attack that peaked at 665 Gigabits of traffic per second. It was targeted not at a huge multinational org or a government, but at well-known security blogger Brian Krebs.
It’s emotionally damaging to be the victim of a DDoS, to say the least. The technological consequences of an unmitigated DDoS attack can cripple businesses, but the emotional impacts are often ignored. I’ll never forget my first run-in with a DDoS. It started on May 2, 2006 at 4 PM. I was working as an operations architect at Six Apart, a provider of blogging software. We got hit with a 160 Gigabyte-per-second DDoS (which was big at that time) by spammers who were targeting an anti-spam tool we used called Blue Frog. We completely lost all production access from outside. At first we couldn’t believe we were under attack; we were in total denial. For 12 hours we were panicked, but managed to finally get the service working again and mitigate the attack.
The 5 emotional stages of DDoS attacks
Businesses who are victims of DDoS lose revenue during attacks, and their reputations can be damaged. All of that pressure trickles down to the operations engineers who are the first responders to attacks. They are in charge of trying to stop the deluge that is shutting down the company’s online operations. They are in emergency mode with no sleep for days and a non-stop dose of cortisol — the fight or flight chemical that arises during stressful situations. It all takes its toll.
As a CDN, Fastly defends some of the biggest names in the industry against attacks on a daily basis, so we intimately understand what DDoS targets go through. I liken it to the emotional stages that happen during other intense experiences. Here is my list of the five emotional stages of a DDoS attack:
- Hunger. People become hungry and forget to eat – everyone is working hard for a long time in a high adrenaline environment.
- Sadness and worry: this is not like normal outages. There’s no timeline; no one actually knows when it’s going to end. People get scared, and sometimes believe it’s an existential threat to the organization.
- Confusion. There’s no clear reason behind a DDoS — no one tripped a cable; no process failed.
- Anger. A DDoS is not actually blameless, despite popular opinion. It’s not internal; it’s caused because there’s a jerk attacking you.
- Coming to terms. Your entire organization (not just the tech team) is under attack. It’s not a process failure, and it’s not your fault.
So, what do you do to help minimize the emotional turmoil in the case of an attack? You have to be prepared, physically and mentally. A year ago during a DDoS, our VP of infrastructure asked me, “What happens if this is the new normal? We can’t deploy or make changes because half the engineering team is trying to figure out how we mitigate. People are burning out.” If an attack lasts two months, how will you actually operate? You need to plan for that possibility now. Here’s how:
Find the weak spots in your architecture. The attack doesn’t have to be huge — if I can find a page on your website that does a thousand database calls, and I hit that page, your website goes down. And I don’t need many resources to do that. If I hit your front page and it’s perfectly cached, I can do that 100,000 times a second and nothing happens, but you’ll need to find the weak points to prepare in advance for whatever comes your way.
Run drills to train your teams on how they should react. We often get calls from people who are getting threatened with DDoS, and they’re very emotional because they’re scared. If you run drills, you at least know that you have a predefined set of actions to take.
Model your risk. Model what happens if different parts of your infrastructure are under DDoS.
Have a partner. All of us (CDNs) have a lot of bandwidth, and the sad thing about DDoS is if you have a 10 GB link and I send you 10 GBs plus one byte you’re down. You have a 1 TB link and I send you a TB plus one, you’re down.
Configure (and test) the system. Don’t make the same mistake we made in 2006, where we had a mitigation product but we hadn’t configured it (or even remembered that we had it). Test it — you’ll sleep better.
Take care of your people. You need to plan for team sleep and food schedules — the rotation of people becomes even more important because you don’t have a timeline. After eight hours, people need to sleep because otherwise 44 hours later they’re not going to function. Force people offline to rest, and spend a lot of time explaining to the organization what’s going on — much more so than you would for a normal event.
And then, don’t panic. Things will be OK. Attackers will go away. They can’t sustain it forever; they burn resources while they’re attacking you. Understanding the impact — emotional and financial — a DDoS can have on your team and your business can help you better prepare for it.
You may also like:
Google Chrome’s Alex Russell on service workers, PWAs, and what’s next for mobile
At Altitude 2016, Software Engineer Alex Russell discussed the latest projects the “performance obsessed” Google Chrome team had underway. In this recap, we’ll take a look at how you can provide reliable offline experiences, how…
Building and scaling the Fastly network, part 2: balancing requests
In part 1, we discussed how Fastly started down the slippery slope of network software. Our previous experience with routing suggested that avoiding traditional network devices would not only dramatically cut capital expenditure, but…
How to solve anything in VCL, part 3: authentication and feature flags at the edge
In “How to solve anything” parts 1 and 2, we outlined how to use Varnish Configuration Language (VCL) to address some of your more challenging problems. In this post, we’ll discuss how Andrew...