---
title: Managing bot traffic
summary: null
url: >-
https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic
---
You can use the Bot Management controls to manage bot traffic for a service. The [Protection mode setting](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#selecting-a-protection-mode) determines whether Fastly monitors traffic and enforces your protection decisions, while the [bot category and identity controls](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#selecting-protection-for-bot-categories-and-identities) let you specify what action to take for each bot category or identity (e.g., block headless bots but allow search engines).
For custom logic beyond what the Bot Management controls support, you can [create VCL snippets](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#using-vcl-to-manage-bot-traffic) that reference the bot detection variables. For example, you could intentionally serve slightly modified or stale pricing data to SEO and marketing bots.
## Prerequisites
Before configuring bot protection, you must:
- purchase [Bot Management](https://docs.fastly.com/products/bot-management).
- [deploy Bot Management](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/#deploying-bot-management) using pre-cache inspection on each Fastly service where you intend to use ContentGuard.
## Selecting a Protection mode
The Protection mode setting controls whether Fastly monitors your traffic for bots and whether it enforces your [protection decisions](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#selecting-protection-for-bot-categories-and-identities) (e.g., block headless bots). To change the protection mode for a service, complete the following steps:
1. Log in to the [Fastly control panel](https://manage.fastly.com).
2. Go to **Security** > **Bot Management** > **Settings**.
3. From the services menu, select the appropriate service.
4. Select a protection mode. Options include:
- **Block (Active enforcement):** Fastly monitors your traffic for bots and enforces your protection decisions.
- **Log (Monitor only):** Fastly monitors your traffic for bots and logs results. Use this option to observe bot activity before enabling enforcement.
- **Off:** Fastly doesn't monitor, log, or block traffic.
## Selecting protection for bot categories and identities
You can specify what action to take for each bot category or identity from the **Protection** page:
1. Log in to the [Fastly control panel](https://manage.fastly.com).
2. Go to **Security** > **Bot Management** > **Protection**.
3. From the services menu, select the appropriate service.
4. In the row of the bot category that you'd like to configure protection for, do one of the following:
- From the **Action** menu, select the action Fastly should take for all bots that fall into that category.
- Click **Configure bots** and then from the **Action** menu for each bot identity, select the action Fastly should take for all requests from that specific bot. When you change the value for a bot identity, the value of the **Action** menu for the category automatically changes to **Custom**.
> **IMPORTANT:** The **Protection mode** must be set to **Block** for the bot category and identity actions to be enforced.
## Using VCL to manage bot traffic
To implement custom logic beyond what the Bot Management controls offer, create VCL snippets that reference the [bot VCL variables](https://www.fastly.com/documentation/reference/vcl/variables/). For example, you could serve modified content to marketing bots, route traffic to specific origins, or generate custom response pages.
> **HINT:** VCL logic executes after Bot Management applies your bot category and identity actions. Bots blocked by those actions won't reach your cache layer or VCL snippets. For more information, check out our [Edge phases processing model](https://www.fastly.com/documentation/guides/getting-started/concepts/edge-processing-phases-model) guide.
### Controlling bot activity with VCL snippets
To create a [VCL Snippet](https://www.fastly.com/documentation/guides/full-site-delivery/fastly-vcl/vcl-snippets/using-vcl-snippets) in the `vcl_recv` subroutine that customizes bot interactions with your ecosystem, complete the following steps:
1. Log in to the [Fastly control panel](https://manage.fastly.com).
2. From the [**Home**](https://manage.fastly.com/home) page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click **Edit configuration** and then select the option to clone the active version.
4. Click **VCL**.
5. Click **VCL Snippets**.
6. Click **Add snippet**.
7. Fill out the **Add VCL snippet** fields as follows:
- Using the **Type** controls, select **Regular** to create a regular VCL snippet.
- In the **Name** field, enter an appropriate name (e.g., `Bot logic`).
- Using the **Placement** controls, select **Within subroutine**.
- From the **Subroutine** menu, select **recv (`vcl_recv`)**.
- In the VCL editor, add logic that references the [bot VCL variables](https://www.fastly.com/documentation/reference/vcl/variables/), which all start with `fastly.bot.`. For example, this code blocks deceitful user agents, manages AI crawlers, and routes verified search engines:
```vcl
# --- START BOT LOGIC ---
if (fastly.bot.analyzed) {
# Block bots with deceitful User-Agent strings
if (fastly.bot.name == "Deceitful User-Agent") {
error 403 "Forbidden";
}
# Redirect AI crawlers to the licensing requirement
if (fastly.bot.category.is_ai_crawler || fastly.bot.category.is_ai_fetcher) {
error 402 "AI Licensing Required";
}
# Block unverified bots claiming to be legitimate search engines
if (fastly.bot.detected && !fastly.bot.category.is_verified) {
if (req.http.User-Agent ~ "(?i)(Googlebot|bingbot|msnbot|Yandex|Applebot|Baiduspider|Amazonbot|LinkedInBot|Yahoo! Slurp|DuckDuckBot)") {
error 403 "Forbidden";
}
}
# Block specific search engines based on name or region
if (fastly.bot.category.is_search_engine_crawler) {
if (fastly.bot.name ~ "Yandex") {
error 403 "Service restricted in this region.";
}
# Route verified search engines to a static-optimized origin
# In the next line, be sure to delete the hash symbol (`#`) and replace `static_origin_name` with the name of your static backend.
# set req.backend = static_origin_name;
set req.http.X-Bot-Priority = "High";
}
# For SEO and marketing bots, flag the request and serve slightly modified or "stale" pricing data
if (fastly.bot.category.is_search_engine_optimization || fastly.bot.category.is_online_marketing) {
set req.http.X-Serve-Deception-Data = "1";
}
# Generate preview pages for social platforms at the edge
if (fastly.bot.category.is_page_preview) {
error 705 "Social-Preview";
}
# Pass bot name to origin for analytics
if (fastly.bot.detected) {
set req.http.X-Bot-Name = fastly.bot.name;
}
}
```
> **HINT:** This logic triggers error codes (like 402 and 705) for certain bot behaviors. To present custom messaging or redirects to blocked bots based on these error codes, create a [corresponding VCL snippet](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#generating-custom-response-pages) for the `vcl_error` subroutine.
8. Click **Add** to create the snippet.
9. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes.
### Generating custom response pages
To handle the error codes that may be triggered by your [VCL logic](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/managing-bot-traffic#controlling-bot-activity-with-vcl-snippets) that controls bot activity, create a [VCL snippet](https://www.fastly.com/documentation/guides/full-site-delivery/fastly-vcl/vcl-snippets/using-vcl-snippets) for the `vcl_error` subroutine. This allows you to present custom messaging or redirects to blocked bots based on the specific error codes.
1. Log in to the [Fastly control panel](https://manage.fastly.com).
2. From the [**Home**](https://manage.fastly.com/home) page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click **Edit configuration** and then select the option to clone the active version.
4. Click **VCL**.
5. Click **VCL Snippets**.
6. Click **Add snippet**.
7. Fill out the **Add VCL snippet** fields as follows:
- Using the **Type** controls, select **Regular** to create a regular VCL snippet.
- In the **Name** field, enter an appropriate name (e.g., `Bot logic`).
- Using the **Placement** controls, select **Within subroutine**.
- From the **Subroutine** menu, select **recv (`vcl_error`)**.
- In the VCL editor, add logic to handle custom error statuses. For example, this code generates response pages for AI licensing requirements and social media previews:
```vcl
# Handle AI licensing requirement (error 402)
if (obj.status == 402) {
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"Commercial License Required
Automated access for AI training requires a commercial agreement.
"};
return(deliver);
}
# Generate social media preview page (error 705)
if (obj.status == 705) {
set obj.status = 200;
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"
Redirecting to full site..."};
return(deliver);
}
```
8. Click **Add** to create the snippet.
9. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes.
## Related content
- [About ContentGuard](https://www.fastly.com/documentation/guides/security/bot-management/contentguard/about-contentguard)
- [Bot VCL variables](https://www.fastly.com/documentation/reference/vcl/variables/)