From AI Crawlers to Headless Bots: How Automated Traffic is Changing the Web

For years, bots have been part of the fabric of the internet. Some help users find content. Others quietly power integrations and workflows from behind the scenes. And some, of course, cause real harm.

What’s changed is how much of the web they now touch – and how much influence they have over business outcomes. In Fastly’s latest Threat Insights Report, we analyzed trillions of application and API requests across our global network to understand how automated traffic is evolving. The headline is hard to ignore: nearly one-third of all web traffic is now bot-driven, and the majority of that traffic isn’t something organizations necessarily want.

“Good” Bots Aren’t Automatically Good for Business

The old model was simple: search crawlers and fetchers were helpful, and everything else was the problem. That distinction doesn’t hold up anymore. Bots have grown up. They’re faster, harder to spot, and increasingly tied to revenue, risk, and reputation.

In Q3, we saw billions of requests from so-called “wanted” bots blocked, accounting for nearly 4% of all bot traffic. This wasn’t accidental. Organizations are taking a closer look at how AI crawlers and fetchers interact with their content – and whether the value exchange still makes sense.

This is especially visible in Media and Entertainment, where publishers are seeing AI systems scrape content at scale while referral traffic keeps shrinking. When AI-generated summaries answer user questions directly, and fewer than 1% of users click through to the source, it forces a hard conversation about access, attribution, and revenue. 

The takeaway isn’t “block all bots.” It’s that intent matters, and blanket allowances no longer reflect how the web actually works.

Headless Bots are Hitting Transactions First

While AI crawlers tend to spark the most discussion, headless bots remain one of the most serious operational threats we see.

These bots look human, behave like browsers, and operate at machine speed. In Q3, 89% of headless bot traffic targeted the Financial and Commerce industries, where logins, checkouts, and APIs directly translate to money moving.

That creates a familiar tension. Many legitimate tools also rely on headless automation, from testing frameworks to internal services. Blocking too aggressively can break workflows. Blocking too loosely can invite fraud, abuse, or downtime. 

The organizations handling this well aren’t relying on simple rules or static allowlists. They’re investing in signals and context that let them tell the difference between harmful automation and the traffic their business actually depends on.

AI is Reshaping Who Crawls What, and Why

The rise of generative AI has changed traffic patterns in ways that are becoming hard to ignore.

In Q3, the Commerce and Public Sector sites received the highest distribution of crawler traffic, while Education saw the largest share of fetcher activity. A small number of AI players accounted for the majority of requests in their categories, reflecting how concentrated this ecosystem already is.

For Commerce, the demand is clear: real-time pricing and inventory data. For publishers, educators, and public institutions, “fresh” content remains a prime target. These shifts aren’t inherently bad – but they do require context and oversight.

Automation now plays a role in discoverability, competitiveness, and control. Treating it as a purely technical concern and not a strategic business decision leaves too much on the table.

Bot Management is a Business Decision Now

One of the clearest signals from the data is that bot traffic can’t be managed in a vacuum anymore.

Bots affect:

• Revenue through referrals, fraud, and transaction integrity

• Infrastructure costs, through sheer volume

• Brand visibility, especially in AI-driven search and discovery

• Innovation, by enabling or limiting how services interact

Understanding bot behavior is no longer just about defense. It’s about making smarter decisions that protect your systems while still allowing your business to move fast.

Read the Full Threat Insights Report

This post only scratches the surface of what we’re seeing across Fastly’s global network. The full Threat Insights Report dives deeper into bot behavior by industry, traffic type, and intent – with real-world data you can use to make smarter security and business decisions.

If automated traffic is shaping your security posture, your performance, or your growth strategy, this is a conversation worth having now. Download the full report.