Introducing ContentGuard: protecting your most valuable IP
Over the last decade, organizations adopting modern architectures in an increasingly global economy have relied on content delivery networks to serve users quickly and cost-effectively by reducing origin fetches. These benefits encouraged DevOps and platform engineering teams to adopt a “cache-everything” mentality, optimizing for cache hits, with many Fastly customers now achieving ratios above 95%. But as we move into the era of AI, this mantra has created the need for organizations to not just optimize for cache, but also understand who… or what, is accessing that content.
Fastly’s latest Security Research report found that nearly half of requests to cached content are from bots, with 99% classified as unverified or malicious. These findings reinforce the need for bot detection in front of cached content across any network. Today, we’re introducing ContentGuard, a new Fastly Bot Management feature that allows customers to control who can access their most valuable intellectual property (IP).
Updating Operations for the AI Era
With so much content cached (and nearly half of requests coming from bots) organizations must control access or risk wasted resources, lost authority, and exposure to unwanted automation from competitors or bad actors. Bots aren’t traditional threats – without advanced detection, their requests often appear legitimate until malicious behavior is detected by downstream security layers. As a result, bots often bypass immediate security concerns while creating longer-term business risks that can lead to significant revenue impact.
Mitigate malicious and deceitful bots wasting resources
Fastly research shows that 47% of traffic interacting with cached content consists of unverified or malicious bots. These represent bots engaging in nefarious activities, falsely impersonating verifiable bots, or operating with incomplete information, signaling that they’re likely not there for legitimate purposes. These bots are often those organizations want to mitigate or, at the very least, gain visibility into as their significant presence not only has security implications (why are they here, what are they probing for, etc.), but also direct operational budget impacts as each request ultimately comes with a cost. ContentGuard makes it easy to track and respond to these often-unwanted bots with simple rules that work in perpetuity, rather than solutions that require regular tuning.
Control AI Crawlers and Fetchers Scraping Content
Verifiable bots include Crawlers, Fetchers, and other often “wanted” bots that can have an outsized impact on businesses. While these agents represent a small portion of traffic (~1%), each scrape may represent many real users who no longer visit your site directly, instead receiving answers from large language models. This one-to-many relationship is taking center stage in industries like digital publishing, advertising, and others where user visits drive impressions, subscriptions, and revenue. By implementing ContentGuard, these currently anonymous requests are clearly labeled as the verifiable crawlers, fetchers, etc. that they are. This visibility allows teams to decide:
Which “wanted” bots are truly beneficial to your business
When to allow full access, limit content exposure, or block specific agents
How to treat verified bots — rate limit some, monetize others, or block certain activity entirely
Take control of your most valuable digital assets
The nuanced nature of bots (especially in light of modern infrastructure) necessitates that organizations consider where bots can impact their business and knowingly adapt their strategy accordingly. ContentGuard makes it easy to identify which bots are accessing your most valuable content and how they interact with it. Whether you’re an enterprise trying to reduce your operational costs and prevent malicious bots from stealing content, a publisher trying to take control of what AI crawlers and fetchers have access to, or anything in between, Fastly is making it simple to get the protection and control needed to operate in the era of AI. Built as a feature within Fastly Bot Management, our solution helps you respond to automated traffic with the nuance your business and security posture requires. With clear visibility, the ability to go beyond simple blocking, and tools to shape how bots interact with your business, it’s the trusted solution by many of the world’s largest brands. Contact us to learn more about Bot Management and ContentGuard.