How to Control and Monetize AI Bot Traffic Using Fastly and TollBit

AI bots are taking over the web, accounting for over a third of all traffic in Q2. While the conversation often focuses on how to stop unwanted bots, those attempting Account Takeovers, launching DDoS attacks, and so on, publishers, ecommerce companies, and other content providers have been fighting a much more nuanced battle. 

As generative AI models grow more capable and hungrier for our data – publishers, content creators, and digital platforms are all asking the same question: how do we protect our content from being taken without our consent and compensation?

Fastly and TollBit have partnered to deliver a powerful integration that enhances bot management and opens new monetization opportunities for organizations. This collaboration combines Fastly's advanced bot detection capabilities with TollBit's robust verification and analytics platform, enabling customers to gain control, consent, and compensation for AI fetching and crawling.

Meet the Fastly + TollBit Integration

Earlier this year, Fastly introduced AI Bot Management. It gives teams the power to manage and control the behavior of AI bots that crawl and scrape website content. With AI Bot Management, you can detect which AI bots are accessing your content and take action to block, allow, or intercept particular AI bots based on your own unique policies and desired responses. 

Through a seamless new integration with TollBit, publishers, ecommerce, and other content providers gain an efficient way to manage and monetize bot traffic. By combining Fastly’s world-class bot management with TollBit’s powerful verification, analytics, and monetization platform, organizations have the ability to:

• Detect and classify bots via Fastly AI Bot Management

• Redirect AI bots to your TollBit Bot Paywall (a subdomain you set up for free)

• Verify tokens and grant content access to bots with valid tokens

• Require all other AI bots to pay for access

Key Features

• Customizable Bot Detection: Fastly AI Bot Management offers customizable bot detection beyond basic user agent checking, allowing publishers to classify bots as good or bad based on their needs.

• Flexible Enforcement Levels: Fastly AI Bot Management allows teams to select enforcement levels for both suspected and verified bots, ensuring that only unwanted traffic is redirected.

• Verification and Analytics: TollBit verifies authorized AI bot requests and provides a comprehensive dashboard with AI bot analytics, focusing on legitimate AI access.

Implementing a real example

Let's walk through setting up AI bot monetization using a practical example. Suppose you want to monetize traffic from ChatGPT, Claude, and Perplexity instead of blocking them.

Starting Point - Your robots.txt:

User-agent: GPTBot
User-agent: ChatGPT-User  
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /

Part 1: Fastly AI Bot Management Setup

Configure Redirect Rules:

1. In your Fastly AI Bot Management dashboard, create a new redirect rule.

2. Add the specific AI bot user agents you want to monetize (GPTBot, ClaudeBot, PerplexityBot).

3. Set the redirect destination to your TollBit subdomain (e.g., tollbit.yoursite.com).

4. Choose enforcement level - "verified bots" for higher confidence detection.

5. Deploy the configuration.

How the redirect works:

AI bot visits your site → Fastly detects and classifies the bot → Instead of blocking, bot gets redirected to your TollBit Bot Paywall → AI companies with valid tokens get content access → Others are directed to pay for access.

Part 2: TollBit Setup

1. Create Your TollBit Account & Subdomain: Publishers can sign up for free at TollBit and set up a subdomain (e.g., tollbit.yoursite.com). This subdomain serves as the gateway for AI bots and agents.

2. Configure Monetization Rules: Set up your pricing and access rules for different AI companies. For detailed configuration options, see our documentation.

That's it – your AI bot traffic is now being monetized – for free!

Stop Scrapers from Stealing Your Valuable Content

The entire process leverages your existing Fastly infrastructure with no additional latency, seamless to human visitors.

The Fastly + TollBit integration provides:

• Granular control: Decide which bots to monetize vs block

• Performance: Built on Fastly’s edge infrastructure

• Flexibility: Full customization of bot classification and routing

• Familiar interface: Use your existing AI Bot Management dashboard

This integration represents a significant step forward in bot management and monetization. By leveraging Fastly’s customizable bot detection and TollBit’s verification and analytics, publishers can effectively manage bot traffic and unlock new revenue streams.

Fastly and TollBit will continue to evolve this integration, with an eye toward making AI bot enforcement and analytics workflows more efficient to serve the publishing, ecommerce, and other content-rich communities even more in the future. We’d love to hear your feedback on how we can keep building a safer, fairer internet together.

Get Started Today

This integration uses capabilities already available in Fastly AI Bot Management. If you're interested in monetizing AI bot traffic:

1. Sign up for TollBit at tollbit.com. Reach out to team@tollbit.com for assistance or questions.

2. Follow our integration documentation for step-by-step redirect configuration

3. Start monetizing your AI bot traffic within minutes

Original content is hard to come by, and yet, AI bots are scraping that content for free and redistributing it to anyone leveraging their chatbot of choice (Gemini, Claude, ChatGPT, etc.). With just a simple update to your redirect rules in Fastly AI Bot Management, you can move beyond blocking bots and instead gain visibility, control, and even compensation, thanks to the combined power of Fastly and TollBit. 

The next evolution of bot management is here and ready to deploy. The first step to leveraging this integration is getting started with Fastly Bot Management. Get in touch and bring an end to unmetered scraping.