Managing bot traffic

You can use the Bot Management controls to manage bot traffic for a service. The Protection mode setting determines whether Fastly monitors traffic and enforces your protection decisions, while the bot category and identity controls let you specify what action to take for each bot category or identity (e.g., block headless bots but allow search engines).

For custom logic beyond what the Bot Management controls support, you can create VCL snippets that reference the bot detection variables. For example, you could intentionally serve slightly modified or stale pricing data to SEO and marketing bots.

Prerequisites

Before configuring bot protection, you must:

• purchase Bot Management.
• deploy Bot Management using pre-cache inspection on each Fastly service where you intend to use ContentGuard.

Selecting a Protection mode

The Protection mode setting controls whether Fastly monitors your traffic for bots and whether it enforces your protection decisions (e.g., block headless bots). To change the protection mode for a service, complete the following steps:

1. Log in to the Fastly control panel.

2. Go to Security > Bot Management > Settings.

3. From the services menu, select the appropriate service.

4. Select a protection mode. Options include:

   • Block (Active enforcement): Fastly monitors your traffic for bots and enforces your protection decisions.
   • Log (Monitor only): Fastly monitors your traffic for bots and logs results. Use this option to observe bot activity before enabling enforcement.
   • Off: Fastly doesn't monitor, log, or block traffic.

Selecting protection for bot categories and identities

You can specify what action to take for each bot category or identity from the Protection page:

1. Log in to the Fastly control panel.
2. Go to Security > Bot Management > Protection.
3. From the services menu, select the appropriate service.
4. In the row of the bot category that you'd like to configure protection for, do one of the following:
   • From the Action menu, select the action Fastly should take for all bots that fall into that category.
   • Click Configure bots and then from the Action menu for each bot identity, select the action Fastly should take for all requests from that specific bot. When you change the value for a bot identity, the value of the Action menu for the category automatically changes to Custom.

Using VCL to manage bot traffic

To implement custom logic beyond what the Bot Management controls offer, create VCL snippets that reference the bot VCL variables. For example, you could serve modified content to marketing bots, route traffic to specific origins, or generate custom response pages.

Controlling bot activity with VCL snippets

To create a VCL Snippet in the vcl_recv subroutine that customizes bot interactions with your ecosystem, complete the following steps:

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.

4. Click VCL.

5. Click VCL Snippets.

6. Click Add snippet.

7. Fill out the Add VCL snippet fields as follows:

   • Using the Type controls, select Regular to create a regular VCL snippet.

   • In the Name field, enter an appropriate name (e.g., Bot logic).

   • Using the Placement controls, select Within subroutine.

   • From the Subroutine menu, select recv (vcl_recv).

   • In the VCL editor, add logic that references the bot VCL variables, which all start with fastly.bot.. For example, this code blocks deceitful user agents, manages AI crawlers, and routes verified search engines:

     1
     2
     3
     4
     5
     6
     7
     8
     9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46

     # --- START BOT LOGIC ---
     if (fastly.bot.analyzed) {
        # Block bots with deceitful User-Agent strings
        if (fastly.bot.name == "Deceitful User-Agent") {
           error 403 "Forbidden";
        }
     
        # Redirect AI crawlers to the licensing requirement
        if (fastly.bot.category.is_ai_crawler || fastly.bot.category.is_ai_fetcher) {
           error 402 "AI Licensing Required";
        }
     
        # Block unverified bots claiming to be legitimate search engines
        if (fastly.bot.detected && !fastly.bot.category.is_verified) {
           if (req.http.User-Agent ~ "(?i)(Googlebot|bingbot|msnbot|Yandex|Applebot|Baiduspider|Amazonbot|LinkedInBot|Yahoo! Slurp|DuckDuckBot)") {
              error 403 "Forbidden";
           }
        }
     
        # Block specific search engines based on name or region
        if (fastly.bot.category.is_search_engine_crawler) {
           if (fastly.bot.name ~ "Yandex") {
              error 403 "Service restricted in this region.";
           }
     
           # Route verified search engines to a static-optimized origin
           # In the next line, be sure to delete the hash symbol (`#`) and replace `static_origin_name` with the name of your static backend.
           # set req.backend = static_origin_name;
           set req.http.X-Bot-Priority = "High";
        }
     
        # For SEO and marketing bots, flag the request and serve slightly modified or "stale" pricing data
        if (fastly.bot.category.is_search_engine_optimization || fastly.bot.category.is_online_marketing) {
           set req.http.X-Serve-Deception-Data = "1";
        }
     
        # Generate preview pages for social platforms at the edge
        if (fastly.bot.category.is_page_preview) {
           error 705 "Social-Preview";
        }
     
        # Pass bot name to origin for analytics
        if (fastly.bot.detected) {
           set req.http.X-Bot-Name = fastly.bot.name;
        }
     }

     HINT: This logic triggers error codes (like 402 and 705) for certain bot behaviors. To present custom messaging or redirects to blocked bots based on these error codes, create a corresponding VCL snippet for the vcl_error subroutine.

8. Click Add to create the snippet.

9. From the Activate menu, select Activate on Production to deploy your configuration changes.

Generating custom response pages

To handle the error codes that may be triggered by your VCL logic that controls bot activity, create a VCL snippet for the vcl_error subroutine. This allows you to present custom messaging or redirects to blocked bots based on the specific error codes.

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.

4. Click VCL.

5. Click VCL Snippets.

6. Click Add snippet.

7. Fill out the Add VCL snippet fields as follows:

   • Using the Type controls, select Regular to create a regular VCL snippet.

   • In the Name field, enter an appropriate name (e.g., Bot logic).

   • Using the Placement controls, select Within subroutine.

   • From the Subroutine menu, select recv (vcl_error).

   • In the VCL editor, add logic to handle custom error statuses. For example, this code generates response pages for AI licensing requirements and social media previews:

     1
     2
     3
     4
     5
     6
     7
     8
     9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19

     # Handle AI licensing requirement (error 402)
     if (obj.status == 402) {
        set obj.http.Content-Type = "text/html; charset=utf-8";
        synthetic {"<html><body><h1>Commercial License Required</h1>
        <p>Automated access for AI training requires a commercial agreement.</p>
        </body></html>"};
        return(deliver);
     }
     
     # Generate social media preview page (error 705)
     if (obj.status == 705) {
        set obj.status = 200;
        set obj.http.Content-Type = "text/html; charset=utf-8";
        synthetic {"<html><head>
           <meta property="og:title" content="Page Preview" />
           <meta property="og:description" content="View this content on our website." />
        </head><body>Redirecting to full site...</body></html>"};
        return(deliver);
     }

8. Click Add to create the snippet.

9. From the Activate menu, select Activate on Production to deploy your configuration changes.

Related content

• About ContentGuard
• Bot VCL variables