---
title: How AI Runtime Control works
summary: >-
  AI Runtime Control (ARC) routes your application's requests to the LLM
  providers it uses, giving you one place to manage provider credentials, issue
  virtual keys, and monitor AI use.
url: >-
  https://www.fastly.com/documentation/guides/platform/ai-runtime-control/how-ai-runtime-control-works
---

AI Runtime Control (ARC) is a control plane that sits between your application and the large language model (LLM) providers it uses. Your application sends requests to ARC instead of calling providers directly. ARC forwards each request to the provider and model you configured it for, returns the provider's response, and records the request.

Because requests pass through ARC, usage can be attributed to the virtual key that made each request, limits can be applied before a request reaches a provider, and requests can be inspected for things like prompt injection attempts.

## What happens to a request

![A horizontal flow diagram. On the left, a box labeled "Your application" with an arrow to the right labeled "request with virtual key". In the middle, a box labeled "AI Runtime Control (ARC)" containing three stacked steps: "Identify virtual key and model", "Apply limits", "Substitute provider credentials". An arrow continues right to a box labeled "Your LLM provider". A return arrow runs back from the provider through ARC to the application, labeled "response".](/img/arc-request-flow.png)

Your application sends every request to the same ARC endpoint, regardless of which provider or model it's destined for. It authenticates with a _virtual key_, an ARC-issued credential that stands in for a raw provider key. Your provider credentials stay in ARC, so your application never holds them.

ARC identifies the virtual key and determines which provider and model the request is destined for. A _provider_ is an LLM service that ARC routes to (for example, Anthropic). You bring your own account and credentials for each provider you use, and you can also [register a model endpoint you host yourself](https://www.fastly.com/documentation/guides/platform/ai-runtime-control/managing-providers/). ARC does not host or run models.

ARC then applies the limits configured for that virtual key, substitutes your provider credentials, and forwards the request. When the provider responds, ARC returns the response to your application and records the request.

> **NOTE:** 
>
> Fastly does not provide direct support for third-party services. Read [Fastly's Terms of Service](https://www.fastly.com/terms) for more information.
>
>

## Virtual keys

A virtual key carries the routing and the limits that apply to every request sent with it.

- **Each key maps to an ordered list of providers.** Check out the section on [failover](https://www.fastly.com/documentation/guides/platform/ai-runtime-control/how-ai-runtime-control-works#failover) for how ARC uses that list.
- **Keys are separate, so usage is separable.** You can issue a key per application, per environment, or per developer, and see usage attributed to each one.
- **You can refresh or delete a key at any time.** Refreshing changes the credential value and deleting revokes access, without touching your provider account. In both cases, requests already in flight complete normally, and new requests using the old value are rejected.

Check out our guidance on how to [manage virtual keys](https://www.fastly.com/documentation/guides/platform/ai-runtime-control/managing-virtual-keys/).

## Failover

ARC sends each request to the first provider in your virtual key's list. If that provider can't serve the request, ARC tries the next one, and so on down the list. Each request starts fresh at the top, so once a provider recovers, requests go back to it without any action from you.

How you arrange that list, and how ARC records what happens when it moves down it, are both worth understanding:

- **You set the order, and you can change it.** The first provider in the list takes your traffic, and the rest are backups in the order you choose. You can reorder the list at any time. A list can hold a single provider, so failover is optional.
- **Attempts are attributed to the provider that served them.** A request that fails over is recorded against the provider that ultimately handled it, so usage and spend reflect where the request actually went.

ARC does not split live traffic across providers. Requests go to your primary provider, and a backup is used only when the provider above it fails. Your application does not need to handle the retry.

## Accounting and logging

A _session_ groups related requests so you can attribute usage to a conversation or workflow rather than to individual calls. Your client supplies the session identifier with the request.

ARC records each request it routes. For every request, it captures:

- the virtual key that made the request
- the session, when the client supplies a session ID
- the provider and model
- the message sent and the response returned
- input and output token counts
- a timestamp

You can investigate these in two views. A [summary view](https://www.fastly.com/documentation/guides/platform/ai-runtime-control/monitoring-ai-usage-with-arc/#viewing-the-arc-summary) presents request volume and token volume, and a [logging view](https://www.fastly.com/documentation/guides/platform/ai-runtime-control/monitoring-ai-usage-with-arc/#viewing-the-logs) presents the request records, searchable and filterable by virtual key, user, provider, model, and date range.

## Limits

You can set limits on a virtual key so that usage stays inside boundaries you choose.

- **Rate limits cap how fast a key can consume.** You can cap a virtual key by requests per minute or by tokens per minute. Token counts are not known until a response completes, so token-based rate limits are applied on a best-effort basis rather than exactly.
- **Budget limits cap spending.** Budget alerts notify you as spending approaches a threshold you set, and you choose what happens when a key reaches its limit, such as alerting only or alerting and blocking.

When a request exceeds a rate limit, ARC rejects it before it reaches your provider.

## Security inspection

> **NOTE:** Security inspection is provided by AI Firewall, a separately purchased add-on. You enable it on each virtual key you want inspected.

Security inspection examines the requests and responses flowing through ARC and identifies prompt injection attempts, which are attacks that embed instructions inside user input to override the system instructions you gave the model. Because a model receives your instructions and untrusted user input as a single stream of text, it has no inherent way to tell them apart.

![A horizontal flow diagram. On the left, a box labeled "Your application" with an arrow to the right labeled "request with virtual key". In the middle, a box labeled "AI Runtime Control (ARC) with AI Firewall Enabled" containing five stacked steps: "Inspect request", "Identify virtual key and model", "Apply limits", "Substitute provider credentials", "Inspect response". An arrow continues right to a box labeled "Your LLM provider". A return arrow runs back from the provider through ARC to the application, labeled "response".](/img/arc-with-firewall-request-flow.png)

Inspection adds checks to the request path, and you control what happens when one of them fires:

- **Inspection happens at two points.** Before ARC forwards a request, AI Firewall checks it against known prompt injection and jailbreak signatures, and wraps untrusted user input in cryptographic boundary tokens with instructions telling the model to treat the wrapped content as data rather than as instructions. After the provider responds and before ARC returns the response to your application, AI Firewall inspects the response for evidence that an injection attempt reached the model.
- **You choose what happens on a detection.** In log mode, ARC records the detection and forwards the request. In block mode, ARC rejects the request before it reaches your provider.
- **Detections appear in the records you already have.** When security inspection is enabled, detection tags, threat classification, and canary status appear alongside the request details ARC already captures.

Structural isolation adds tokens to each request, and your LLM provider bills you for them as part of your normal token usage. Response inspection operates on complete responses, so streaming requests are proxied without response inspection.