Working with surrogate keys

Surrogate keys are unique identifiers that you assign to groups of content for processing. While there are many use cases for surrogate keys, one of the primary way for using them with Fastly is to make purging more efficient.

Surrogate keys allow you to selectively purge related content. Using the Surrogate-Key header, you can "tag" content with a key term, a string of any characters you want. When you want to purge content associated with that key, you then issue a key purge request and all of the objects associated with that key will be purged. This process makes it easier to cache and purge content that changes rapidly and unpredictably without having to purge your entire cache.

Before you begin

We recommend reviewing our guide on adding headers on HTTP requests and responses and working with surrogate keys before purging surrogate keys.

Limitations and considerations

• Surrogate keys must be formatted as a single string without spaces. Spaces separate keys.
• Surrogate keys are case-sensitive.
• Individual tokens may not exceed 1KB in length and Surrogate-Key header values may not exceed 16KB in length. If either of the key or key header value limits are reached while parsing a Surrogate-Key header, the token currently being parsed and all tokens following it within the same header will be ignored.
• The simple cache interface does not support adding surrogate keys.
• Fastly automatically removes any Surrogate-Key headers present on a response before delivering it to the end user (unless Fastly-Debug is included in the request).
• Surrogate key purges are not logged.

Understanding the Surrogate-Key header

To understand how surrogate keys work, especially in the context of content purging, you need to understand what a Surrogate-Key header is and how it behaves.

About the Surrogate-Key header

Any time your origin responds to a request for content, the response it sends will include bits of code called headers in front of the actual body of the response. Those headers are used to give additional detail and provide context for the body of the response itself. HTTP headers sent as part of a response typically look something like this:

HTTP/1.1 200 OK
Content-Type: text/html
Connection: keep-alive
...

You can control that additional information and therefore control how content is served by adding to or modifying the headers your server responds with. The Surrogate-Key header is one of them. It's what categorizes or "tags" the content as part of a specific group. The tag you choose is called a key and it describes the common trait each element of a content group shares.

For example, suppose you have a website where you publish taco recipes. You know that entire categories of recipes will need to be purged, so you thoughtfully include these recipe categories as keys in the Surrogate-Key header. The response your server sends might look more like this:

HTTP/1.1 200 OK
Surrogate-Key: veggie seasonal central-mexico
Content-Type: text/html
...

The addition of the Surrogate-Key header there tells you that the content is tagged with specific identifiers that categorize it for future use. This response contains three surrogate keys: veggie, seasonal, and central-mexico. When Fastly receives a response like this, we use the surrogate keys to create a mapping from each key to the cached content, then we strip out the Surrogate-Key header so it's not included in the response to your readers.

Creating relationships between keys and objects

One of the major advantages of surrogate keys is that they allow for a many-to-many relationship between keys and objects. An origin server's response can associate multiple keys with the object and the same key can be provided in different responses. Take a look at these two requests and responses:

GET /blog/healthy-taco-recipes HTTP/1.1
Host: www.tacolabs.com

HTTP/1.1 200 OK Content-Type: text/html
Content-Length: 1234
Surrogate-Key: mainpage low-carb

GET /recipes/low-carb-cheese-taco-shell HTTP/1.1
Host: www.tacolabs.com

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 2345
Surrogate-Key: low-carb cheese

In this example, there are two objects (/blog/healthy-taco-recipes and /recipes/low-carb-cheese-taco-shell) with three keys (mainpage, low-carb, and cheese). Two of the keys (mainpage and cheese) are associated with a single object and a third key (low-carb) is associated with both objects.

By using the Surrogate-Key header to associate keys with one or more objects, you can precisely control which objects are removed from cache during a purge. Consider the example presented above. Purging the mainpage key would remove only the /blog/healthy-taco-recipes object from the cache. On the other hand, purging the low-carb key would remove both the /blog/healthy-taco-recipes and /recipes/low-carb-cheese-taco-shell objects from the cache.

Wherever there's a relation between two different pieces of content, there might be a good reason to keep them categorized by using a surrogate key.

Generating and setting surrogate keys

There are two ways to set the Surrogate-Key title: by adding the header in the Fastly control panel, or by generating the keys with your own application.

Generating and setting surrogate keys from your application logic

You can make your surrogate key associations on your own application server and include them in the HTTP response that you send to Fastly. This is the more useful method because you can assign exactly the keys that you want on every response:

Surrogate-Key: template-product product-id-724253 product-id-242129 offers

Surrogate keys can also be added explicitly in your edge code. In CDN services, surrogate keys can be added to objects using VCL code in the vcl_fetch subroutine, before they are written to cache:

set beresp.http.Surrogate-Key = "template-product product-id-724253 product-id-242129 offers";

In Compute services, you can specify the desired keys in a cache override settings object attached to the request. Each SDK has a method to do this.

Generating and setting surrogate keys from the Fastly control panel

To set the Surrogate-Key header in the Fastly control panel, follow the steps below:

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.
4. Click Content.

5. Click Create header.

   

6. Fill out the Create a header fields as follows:

   • In the Name field, enter a human-readable name for the header. This name is displayed in the Fastly web interface.
   • From the Type menu, select Cache, and from the Action menu, select Set.
   • In the Destination field, enter http.Surrogate-Key.
   • In the Source field, enter where the content for the header comes from. For example, if you're using Amazon S3 as an origin server, you could set the field to beresp.http.x-amz-meta-surrogate-key to specify x-amz-meta-surrogate-key as the response header received from the backend.
   • From the Ignore if set menu, select No.
   • In the Priority field, enter 10.

7. Click the Create button to create your header.

8. From the Activate menu, select Activate on Production to deploy your configuration changes.

What's next

Now that you understand how the Surrogate-Key header works and how to set it, learn how to purge objects with surrogate keys.

Related content

• Surrogate-Key HTTP header documentation