How we migrated developer.fastly.com from VCL to Compute@Edge

If you build stuff on Fastly, chances are you spend a decent amount of time on our Developer Hub. Last month, we migrated it from our VCL platform to Compute@Edge. Here's how we did it and what you can learn from it.

First, it's important to stress that migrating away from VCL is not necessarily the right thing for you — both our VCL and Compute platforms are actively supported and developed, and VCL offers a fantastic setup experience that will give you great defaults for a fast site with zero code. However, if you want to start doing more complex tasks at the edge, you'll probably need the general purpose compute capabilities of Compute@Edge.

Before you start on the new stuff though, you'll probably want to bring the things you are doing in VCL over to your Compute service. In this post, I’ll look at how to migrate all the patterns used by developer.fastly.com from VCL to Compute@Edge, which makes for a nice example case study.

Groundwork

Since the existing developer.fastly.com service is VCL (and is a production service that we don't want to break!), the first step is to create a new Compute@Edge service that we can test with. This starts with installing the Fastly CLI and initializing a new project:

> fastly compute init
Creating a new Compute@Edge project.
Press ^C at any time to quit.
Name: [edge] developer-hub
Description: Fastly Developer Hub
Author: devrel@fastly.com
Language:
[1] Rust
[2] AssemblyScript (beta)
[3] JavaScript (beta)
[4] Other ('bring your own' Wasm binary)
Choose option: [1] 3
Starter kit:
[1] Default starter for JavaScript
A basic starter kit that demonstrates routing, simple synthetic responses and
overriding caching rules.
https://github.com/fastly/compute-starter-kit-javascript-default
Choose option or paste git URL: [1] 1
✓ Initializing...
✓ Fetching package template...
✓ Updating package manifest...
✓ Initializing package...
Initialized package developer-hub to:
~/repos/Developer-Hub/edge

The Developer Hub is mostly a JavaScript application using the Gatsby framework, so we chose to write our Compute@Edge code in JavaScript too. The app that gets generated by the init command is a complete and working app, so it's worth deploying it to production immediately to get a nice develop-test-iterate cycle going:

> fastly compute publish
✓ Initializing...
✓ Verifying package manifest...
✓ Verifying local javascript toolchain...
✓ Building package using javascript toolchain...
✓ Creating package archive...
SUCCESS: Built package 'developer-hub' (pkg/developer-hub.tar.gz)
There is no Fastly service associated with this package. To connect to an existing service add the Service ID to the fastly.toml file, otherwise follow the prompts to create a service now.
Press ^C at any time to quit.
Create new service: [y/N] y
✓ Initializing...
✓ Creating service...
Domain: [some-funky-words.edgecompute.app]
Backend (hostname or IP address, or leave blank to stop adding backends):
✓ Creating domain some-funky-words.edgecompute.app'...
✓ Uploading package...
✓ Activating version...
Manage this service at:
https://manage.fastly.com/configure/services/*****************
View this service at:
https://some-funky-words.edgecompute.app
SUCCESS: Deployed package (service *****************, version 1)

So now we have a working service, served from the Fastly edge, and can deploy changes to it in a few seconds. Let's start migrating!

Google Cloud Storage

The main content of the Developer Hub is the Gatsby site, which is built and deployed to Google Cloud Storage, then served statically. We can start by adding a backend:

fastly backend create --name gcs --address storage.googleapis.com --version active --autoclone

Now we edit the Compute app's main source file — in this case src/index.js — to load the content from GCS. We can start by replacing the entire content of the file with this:

const BACKENDS = {
GCS: "gcs",
}
const GCS_BUCKET_NAME = "fastly-developer-portal"
async function handleRequest(event) {
const req = event.request
const reqUrl = new URL(req.url)
let backendName
backendName = BACKENDS.GCS
reqUrl.pathname = "/" + GCS_BUCKET_NAME + "/production" + reqUrl.pathname
// Fetch the index page if the request is for a directory
if (reqUrl.pathname.endsWith("/")) {
reqUrl.pathname = reqUrl.pathname + "index.html"
}
const beReq = new Request(reqUrl, req);
let beResp = await fetch(beReq, {
backend: backendName,
cacheOverride: new CacheOverride(["GET", "HEAD", "FASTLYPURGE"].includes(req.method) ? "none" : "pass"),
})
if (backendName === BACKENDS.GCS && beResp.status === 404) {
// Try for a directory index if the original request didn't end in /
if (!reqUrl.pathname.endsWith("/index.html")) {
reqUrl.pathname += "/index.html"
const dirRetryResp = await fetch(new Request(reqUrl, req), { backend: BACKENDS.GCS })
if (dirRetryResp.status === 200) {
const origURL = new URL(req.url) // Copy of original client URL
return createRedirectResponse(origURL.pathname + "/")
}
}
}
return beResp
}
const createRedirectResponse = (dest) =>
new Response("", {
status: 301,
headers: { Location: dest },
})
addEventListener("fetch", (event) => event.respondWith(handleRequest(event)))

This code implements our recommended pattern for fronting a public GCS bucket, but let's step through it:

  • Define some constants. The BACKENDS object is a nice way for us to make the routing extensible for the other backends that will come later. The value "gcs" here matches the name we gave the backend earlier.

  • For convenience, we'll assign the client request event.request to req, and using the URL class we can turn req.url into a parsed URL object, reqUrl.

  • To reach the right object in the GCS bucket, we add the bucket name to the beginning of the path. In our case, we also had to add 'production', since we also store non-production branches in our GCS bucket.

  • If the incoming request ended with /, it makes sense to append 'index.html' to find directory index pages within the bucket.

  • If the response from Google is a 404, and the client request did not end in a slash, we try adding '/index.html' to the path and try again. If it works, we return an external redirect to tell the client to append a slash.

  • At the very end, we attach the request handler function to the client 'fetch' event.

We compile and deploy:

> fastly compute publish
✓ Initializing...
✓ Verifying package manifest...
✓ Verifying local javascript toolchain...
✓ Building package using javascript toolchain...
✓ Creating package archive...
SUCCESS: Built package 'developer-hub' (pkg/developer-hub.tar.gz)
✓ Uploading package...
✓ Activating version...
SUCCESS: Deployed package (service *****************, version 3)

Now, loading some-funky-words.edgecompute.app in a browser, we see the Developer Hub! A great start.

Custom 404 page

Pages that exist now work great, but if we try to load a page that doesn't exist, bad things happen:

404 return

Gatsby does generate a nice 'page not found' page, and deploys it as '404.html' in the GCS bucket. We just need to serve that content if the client request prompted a 404 response from Google:

if (backendName === BACKENDS.GCS && beResp.status === 404) {
// ... existing code here ... //
// Use a custom 404 page
if (!reqUrl.pathname.endsWith("/404.html")) {
debug("Fetch custom 404 page")
const newPath = "/" + GCS_BUCKET_NAME + "/404.html"
beResp = await fetch(new Request(newPath, req), { backend: BACKENDS.GCS })
beResp = new Response(beResp.body, { status: 404, headers: beResp.headers })
}
}

Note that we don't return the beResp we get directly from origin, because it will have a 200 (OK) status code. Instead, we create a new "404" Response using the body stream from the GCS response. This pattern is explored in detail in our backends integration guide.

As before, we run fastly compute publish to deploy, and now our 'page not found' errors start looking a lot nicer:

Custom 404

Redirects

Of course, developer.fastly.com also has a large number of redirects, so those are next up to migrate from VCL. We already use Edge Dictionaries to store them — one for exact redirects and one for prefix redirects. These can be loaded at the start of the request handler:

async function handleRequest(event) {
const req = event.request
const reqUrl = new URL(req.url)
const reqPath = reqUrl.pathname
const dictExactRedirects = new Dictionary("exact_redirects")
const dictPrefixRedirects = new Dictionary("prefix_redirects")
​​ // Exact redirects
const normalizedReqPath = reqPath.replace(/\/$/, "")
const redirDest = dictExactRedirects.get(normalizedReqPath)
if (redirDest) {
return createRedirectResponse(redirDest)
}
// Prefix redirects
let redirSrc = String(normalizedReqPath)
while (redirSrc.includes("/")) {
const redirDest = dictPrefixRedirects.get(redirSrc)
if (redirDest) {
return createRedirectResponse(redirDest + reqPath.slice(redirSrc.length))
}
redirSrc = redirSrc.replace(/\/[^/]*$/, "")
}

The exact redirects are pretty simple, but for the prefix redirects, we need to go around a loop, removing one URL segment at a time until the path is empty, and looking up each progressively shorter prefix in the dictionary. If we find one, then the non-matching part of the client request is appended to the redirect (so a redirect of /source => /destination will redirect a request for /source/foo to /destination/foo).

Canonicalize hostname

At some point, we thought it was a good idea for developer.fastly.com to also be available as fastly.dev. So our VCL service redirects users who request fastly.dev to developer.fastly.com. Let's migrate that next. We start by defining at the very top of the file what the canonical domain for the site is:

const PRIMARY_DOMAIN = "developer.fastly.com"

Then, in the request handler, we read the host header, and redirect if we need to. During testing, we actually set this to the random-funky-words.edgecompute.app-style domain. This deserves to go above the redirects code, just after the declarations at the start of the request handler:

async function handleRequest(event) {
// ... existing code ... //
const hostHeader = req.headers.get("host")
// Canonicalize hostname
if (!req.headers.has("Fastly-FF") && hostHeader !== PRIMARY_DOMAIN) {
return createRedirectResponse("https://" + PRIMARY_DOMAIN + reqPath)
}

Your VCL service might also be performing a TLS redirect for insecure HTTP requests.  Compute@Edge handles this automatically and no longer hands off requests to your code until a secure connection exists — so no need to migrate that.

To test this code, the service needs a second, non-canonical domain attached. We do that with fastly domain create:

fastly domain create --name testing-fastly-devhub.global.ssl.fastly.net --version latest --autoclone

This is a really good use case for Fastly-assigned domains, so you don't need to mess around with DNS (yet). Now if we visit testing-fastly-devhub.global.ssl.fastly.net in the browser, our service redirects us to our primary domain.

Response headers

We then turned to the modifications that our VCL service is making to the responses that we get from GCS. Google adds a bunch of headers to responses that we don't want to expose to the client, like x-goog-generation. Since the backend might add more of these in future, it makes sense to filter the headers based on an allowlist. First, at the top of the file, we define what the allowed response headers are:

const ALLOWED_RESP_HEADERS = [
"cache-control",
"content-encoding",
"content-type",
"date",
"etag",
"vary",
]

Then, just after the backend fetch, we can insert code to filter the headers we get back:

// Filter backend response to retain only allowed headers
beResp.headers.keys().forEach((k) => {
if (!ALLOWED_RESP_HEADERS.includes(k)) beResp.headers.delete(k)
})

Conversely, there are some headers that we do want to see on client responses, like Content-Security-Policy, so we need to add those. Many of these only need to be present on HTML responses:

if ((beResp.headers.get("content-type") || "").includes("text/html")) {
beResp.headers.set("Content-Security-Policy", "default-src 'self'; scrip...")
beResp.headers.set("X-XSS-Protection", "1")
beResp.headers.set("Referrer-Policy", "origin-when-cross-origin")
beResp.headers.append(
"Link",
"</fonts/CircularStd-Book.woff2>; rel=preload; as=font; crossorigin=anonymous," +
"</fonts/Lexia-Regular.woff2>; rel=preload; as=font; crossorigin=anonymous," +
"<https://www.google-analytics.com>; rel=preconnect"
)
beResp.headers.set("alt-svc", `h3-29=":443";ma=86400,h3-27=":443";ma=86400`)
beResp.headers.set("Strict-Transport-Security", "max-age=86400")
}

Adding and removing headers is a very common use case for both VCL and Compute services.

Passthrough backends

Another thing our VCL service does is to pass some requests to backends other than GCS, so that we can invoke cloud functions or other third-party services. As an example, we use Swiftype for our Developer Hub search engine, so to enable the search API to be available within the developer.fastly.com domain, we can add a new backend and then wire up certain request paths to target that backend.

First, we use the Fastly CLI to add the backend:

fastly backend create --name swiftype --address search-api.swiftype.com --version active --autoclone

Then we add a constant to our source code that allows us to reference that backend:

const BACKENDS = {
GCS: "gcs",
SWIFTYPE: "swiftype",
}

Finally, we update the backend selection code to add some routing logic to select Swiftype when appropriate:

let backendName
if (reqPath === "/api/internal/search") {
backendName = BACKENDS.SWIFTYPE
reqUrl.pathname = "/api/v1/public/engines/search.json"
} else {
backendName = BACKENDS.GCS
reqUrl.pathname = "/" + GCS_BUCKET_NAME + "/production" + reqUrl.pathname
// Fetch the index page if the request is for a directory
if (reqUrl.pathname.endsWith("/")) {
reqUrl.pathname = reqUrl.pathname + "index.html"
}
}

You can repeat this for any other backends that you have. We use the same technique for Sentry and Formkeep, for example. We'll be writing about this pattern in more detail soon.

Deployment with GitHub Actions

Our VCL service was not managed in source control, so this is a good opportunity to fix that.  The Compute@Edge version is stored along with the main source of the Developer Hub, allowing us to coordinate edge releases with backend updates and make them dependent so that we don't ship a change to edge logic if we fail to deploy to the backend.

We added two jobs to our CI workflow. One to build the edge app for PRs:

build-fastly:
name: C@E build
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Set up Fastly CLI
uses: fastly/compute-actions/setup@main
- name: Install edge code dependencies
run: cd edge && npm install
- name: Build Compute@Edge Package
uses: fastly/compute-actions/build@main
with:
project_directory: ./edge/

And a second, to deploy the edge app once the GCS deploy has completed:

update-edge-app:
name: Deploy C@E app
runs-on: ubuntu-latest
if: ${{ github.ref_name == 'production' }}
needs: deploy
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Install edge code dependencies
run: cd edge && npm install
- name: Deploy to Compute@Edge
uses: fastly/compute-actions@main
with:
project_directory: ./edge/
env:
FASTLY_API_TOKEN: ${{ secrets.FASTLY_API_TOKEN }}

Next steps

The migration complete, we now have the same features as we had with the VCL service. But now that the Developer Hub is fronted by Compute@Edge, we can move all our cloud functions into our edge compute code directly and start doing more heavy lifting at the edge.

Migrating services to Compute@Edge could be easier than you think. If you're planning your own migration, see our dedicated VCL to Compute@Edge migration guide, which covers many of the patterns we had to convert for the Developer Hub, and our solutions library, which has demos, example code, and tutorials that cover a wide range of use cases.

Not using Compute@Edge yet? Check out how our serverless edge compute platform can help you build faster, more secure, and more performant applications at the edge.

Andrew Betts
Head of Developer Relations
Published
Want to continue the conversation?
Schedule time with an expert
Share this post
Andrew Betts
Head of Developer Relations

Andrew Betts is Head of Developer Relations for Fastly, where he works with developers across the world to help make the web faster, more secure, more reliable, and easier to work with. He founded a web consultancy which was ultimately acquired by the Financial Times, led the team that created the FT’s pioneering HTML5 web app, and founded the FT’s Labs division. He is also an elected member of the W3C Technical Architecture Group, a committee of nine people who guide the development of the World Wide Web.