1. Home
2. Guides
3. Security
4. Bot Management

Unauthorized content scraping of your web applications can lead to intellectual property loss, brand misrepresentation, and server strain. ContentGuard addresses this by detecting and mitigating scraping activity, allowing you to prevent persistent scrapers from harvesting your content.

Prerequisites

Before setting up ContentGuard, you must:

• purchase Bot Management.
• deploy Bot Management using pre-cache inspection on each Fastly service where you intend to use ContentGuard.

Monitoring content scraping

Once you've enabled pre-cache inspection for Bot Management on your service, you can monitor content scraping activity from the Bot overview dashboard, which reflects data from all the requests destined for your service. For more detail, you can set up real-time log streaming and include bot detection variables in your custom log format.

Bot overview dashboard

Use the Bot overview dashboard to monitor the volume and types of bot traffic directed at your service.

To access the dashboard in the Fastly control panel:

1. Log in to the Fastly control panel.
2. Go to Security > Bot Management > Dashboards.
3. (Optional) Use the Service and Time menus to filter the displayed data by specific services and time ranges.

The dashboard includes the following metrics:

• Total bot inspected requests: the total number of requests identified as coming from bots.
• Human traffic percentage: the percentage of inspected traffic identified as human visitors.
• Bot traffic percentage: the percentage of inspected traffic identified as bot activity.
• Bot request traffic: a line graph displaying the volume of requests from suspected or verified bots compared to total requests over time.
• Bot category breakdown: a bar graph displaying bot requests categorized by type, such as scrapers, search engines, and other bot categories.

Managing bot interactions

You can control how bots interact with your web application by writing VCL logic that uses the bot VCL variables and then serving custom response pages to blocked bots.

Controlling bot activity with VCL snippets

To create a VCL Snippet in the vcl_recv subroutine that customizes bot interactions with your ecosystem, complete the following steps:

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.

4. Click VCL.

5. Click VCL Snippets.

6. Click Add snippet.

7. Fill out the Add VCL snippet fields as follows:

   • Using the Type controls, select Regular to create a regular VCL snippet.

   • In the Name field, enter an appropriate name (e.g., Bot logic).

   • Using the Placement controls, select Within subroutine.

   • From the Subroutine menu, select recv (vcl_recv).

   • In the VCL editor, add logic that references the bot VCL variables, which all start with fastly.bot.. For example, this code blocks deceitful user agents, manages AI crawlers, and routes verified search engines:

     # --- START BOT LOGIC ---
     if (fastly.bot.analyzed) {
        # Block bots with deceitful User-Agent strings
        if (fastly.bot.name == "Deceitful User-Agent") {
           error 403 "Forbidden";
        }
     
        # Redirect AI crawlers to the licensing requirement
        if (fastly.bot.category.is_ai_crawler || fastly.bot.category.is_ai_fetcher) {
           error 402 "AI Licensing Required";
        }
     
        # Block unverified bots claiming to be legitimate search engines
        if (fastly.bot.detected && !fastly.bot.category.is_verified) {
           if (req.http.User-Agent ~ "(?i)(Googlebot|bingbot|msnbot|Yandex|Applebot|Baiduspider|Amazonbot|LinkedInBot|Yahoo! Slurp|DuckDuckBot)") {
              error 403 "Forbidden";
           }
        }
     
        # Block specific search engines based on name or region
        if (fastly.bot.category.is_search_engine_crawler) {
           if (fastly.bot.name ~ "Yandex") {
              error 403 "Service restricted in this region.";
           }
     
           # Route verified search engines to a static-optimized origin
           # In the next line, be sure to delete the hash symbol (`#`) and replace `static_origin_name` with the name of your static backend.
           # set req.backend = static_origin_name;
           set req.http.X-Bot-Priority = "High";
        }
     
        # For SEO and marketing bots, flag the request and serve slightly modified or "stale" pricing data
        if (fastly.bot.category.is_search_engine_optimization || fastly.bot.category.is_online_marketing) {
           set req.http.X-Serve-Deception-Data = "1";
        }
     
        # Generate preview pages for social platforms at the edge
        if (fastly.bot.category.is_page_preview) {
           error 705 "Social-Preview";
        }
     
        # Pass bot name to origin for analytics
        if (fastly.bot.detected) {
           set req.http.X-Bot-Name = fastly.bot.name;
        }
     }

     HINT: This logic triggers error codes (like 402 and 705) for certain bot behaviors. To present custom messaging or redirects to blocked bots based on these error codes, create a corresponding VCL snippet for the vcl_error subroutine.

8. Click Add to create the snippet.

9. From the Activate menu, select Activate on Production to deploy your configuration changes.

Generating custom response pages

To handle the error codes that may be triggered by your VCL logic that controls bot activity, create a VCL snippet for the vcl_error subroutine. This allows you to present custom messaging or redirects to blocked bots based on the specific error codes.

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.

4. Click VCL.

5. Click VCL Snippets.

6. Click Add snippet.

7. Fill out the Add VCL snippet fields as follows:

   • Using the Type controls, select Regular to create a regular VCL snippet.

   • In the Name field, enter an appropriate name (e.g., Bot logic).

   • Using the Placement controls, select Within subroutine.

   • From the Subroutine menu, select recv (vcl_error).

   • In the VCL editor, add logic to handle custom error statuses. For example, this code generates response pages for AI licensing requirements and social media previews:

     # Handle AI licensing requirement (error 402)
     if (obj.status == 402) {
        set obj.http.Content-Type = "text/html; charset=utf-8";
        synthetic {"<html><body><h1>Commercial License Required</h1>
        <p>Automated access for AI training requires a commercial agreement.</p>
        </body></html>"};
        return(deliver);
     }
     
     # Generate social media preview page (error 705)
     if (obj.status == 705) {
        set obj.status = 200;
        set obj.http.Content-Type = "text/html; charset=utf-8";
        synthetic {"<html><head>
           <meta property="og:title" content="Page Preview" />
           <meta property="og:description" content="View this content on our website." />
        </head><body>Redirecting to full site...</body></html>"};
        return(deliver);
     }

8. Click Add to create the snippet.

9. From the Activate menu, select Activate on Production to deploy your configuration changes.

Related content

• About Bot Management
• About Fastly's real-time log streaming