You need to fetch data from external APIs and add extra headers with additional useful information to the origin
Requests passing through Fastly can be transformed in many useful ways, and one of the most common is to add information to a request that was not included by the client by appending additional HTTP headers before sending the request on to the backend.
Fastly exposes a variety of information automatically, such as the geolocation and network related information available in Rust via the
Geo interface. This information is simple to add to a client request before passing it to an origin:
let client_ip = req.get_client_ip_addr().unwrap(); let geo = geo_lookup(client_ip).unwrap(); let country_code = geo.country_code(); req.set_header("Fastly-Geo-Country", country_code);
There are also countless data sources that can provide valuable information and add intelligence to your applications. If these sources expose an API, we can query it at the edge to enrich requests with the data that service provides. This might be one of your own services, like an A/B testing API, or a third-party service.
For the purpose of this tutorial, we will detect requests that contain passwords, send an API request to the Have I Been Pwned (HIBP) API to check whether the password has been leaked, and then add a header to the request before it is forwarded to the origin.
"Have I been pwned" (HIBP) is a community service that maintains a database of compromised passwords. It provides an API endpoint that allows passwords to be checked against that database in a privacy-preserving way. This works using a k-anonymity principle, which we can invoke like this:
- Take the original, clear-text credential, such as '123456'.
- Make a SHA1 hash out of the credential, which results in a 40 character string.
- Split the hash into two strings: the first 5 characters, and the remaining 35 characters.
- Send the first 5 characters to the HIBP API.
- HIBP returns a list of all the SHA1 hashes in its database that begin with those 5 characters.
- Use the last 35 characters to determine whether the full hash is in the list.
This mechanism allows a precise trade-off to be made between information leakage and functionality. Let's see how this works using command-line tools:
$ printf '123456' | openssl sha1(stdin)= 7c4a8d09ca3762af61e59520943dc26494f8941b
$ curl https://api.pwnedpasswords.com/range/7c4a8 | grep -i 'd09ca3762af61e59520943dc26494f8941b'D09CA3762AF61E59520943DC26494F8941B:24230577
IMPORTANT: While it's possible to use HIBP anonymously, we strongly recommend using an API key. For production use, go to the HIBP API key page and obtain an API key.
Based on the response, we can tell that the password has been reported compromised 24,230,577 times. So '123456' turns out to be a bad idea for a password.
If you haven't already created a Rust-based Compute project, run fastly compute init in a new directory in your terminal and follow the prompts to provision a new service using the default Rust starter kit and with HTTPBin as a backend:
$ mkdir hibp_enrichment && cd hibp_enrichment$ fastly compute init
Creating a new Compute project.
Press ^C at any time to quit.
✓ Initializing...✓ Fetching package template...✓ Updating package manifest...✓ Initializing package...
This will create some files for you, which you'll need to edit as you go through the rest of the tutorial:
fastly.tomldescribes your project and tells the Fastly CLI where to deploy it to.
Cargo.tomlis the Rust package manifest, where you declare your dependencies.
src/main.rsis the source code of your project. This will have some example code in it, which you can remove.
Create your new Compute service. Make a note of the service ID that is returned from the following command.
$ fastly service create --name my-enrichment-demo
Add the service ID into the fastly.toml file:
# This file describes a Fastly Compute package. To learn more visit:# https://www.fastly.com/documentation/reference/compute/fastly-toml/
authors = [...]description = "Check the HIBP API for compromised passwords and send enriched information to the origin"language = "rust"manifest_version = 1name = "hibp_enrichment"service_id = "[your_service_id]"
You'll need two backends for this demo: the HIBP API and your own origin server that will answer the user's request. We'll use
httpbin.org as a stand in for our backend, but feel free to substitute your own. You can add these using the fastly backend create command:
$ fastly backend create --name=api --address=api.pwnedpasswords.com --port=443 --use-ssl --service-id=[your_service_id] --version=latest$ fastly backend create --name=primary --address=httpbin.org --port=443 --use-ssl --service-id=[your_service_id] --version=latest
Add these dependencies to your Cargo.toml file:
fastly = "0.9.1"sha1_smol = "^1"percent-encoding = "^2.2.0"
Afterwards, import them at the top of
src/main.rs. If you haven't already, remove all the existing content of
main.rs and replace it with the following:
fastly dependency provides the SDK for the Fastly platform;
percent-encoding decodes the percent-encoded password that the user submits in the login form POST; and
sha1 to performs the hashing function necessary to be compatible with the HIBP API.
You created two backends on the Fastly service, called
primary. Since they are referenced as strings in Rust, assigning them to constants will help to avoid typos later:
To find a password in the request, you'll need to be able to parse the request body and extract a field by name. In VCL, we have the
subfield function, but there's no equivalent in Rust so now's a good time to write one:
You'll also need to be able to compute a SHA1 hash of the password for the HIBP API. The SHA1 crate does that but it's helpful to have an easy way to get it as a string:
A Compute program in Rust receives a
Request. Since the objective here is to create an improved (enriched) request, a good signature for the enrichment function would be
Request -> Result<Request, Error>. This enables the enrichment logic to be nicely encapsulated and can be invoked elegantly as part of processing the incoming request, in conjunction with other similar handlers.
The function will therefore take a
Request, add a
Fastly-Password-Status header to it, and then return it to the calling scope.
First, take the request body from the request and use the helper function defined earlier to search it for a credential. For the purposes of this tutorial, we'll assume that the body is
application/x-www-form-urlencoded and that the field name we want is always
password, so a body that would match would be
username=Jo&password=123456. Special characters such as "!" and "$" are often used in passwords, but these special characters will be percent-encoded when a user submits a credential for this tutorial. Therefore, we must decode the percent-encoded password before interacting with the HIBP API.
If there's no credential found in the body, it would be very inefficient to make an unnecessary API request, so in this case, you can create a fast path by adding a
Fastly-Password-Status: no-credential header and returning immediately.
Where a credential is found, the other helper function defined earlier can be used to compute a SHA1 hash as a 40 character string. The HIBP API takes a 5-character prefix of that as an input, so divide the hash into a 5-character
hash_left and a 35-character
hash_right. The request to the API will return a list of 'right hand sides' of all hashes in the database that start with the supplied 'left hand side'. It's then easy enough to check whether
hash_right is in the list, and if so, conclude that the credential is compromised.
The entry point for a Compute program is the
main function. A simple scenario here is to pass every request directly to a backend, and then to return whatever the backend responds with. You need only make a small modification to this - insert a call to the enrichment function, which will modify the request, before you send it to the origin.
You now have a complete Compute program, which receives a
Request, enriches it, forwards it to an origin, and then uses the returned
Response to reply to the client.
Normally, your backend (the
primary backend here) would serve pages that would invite a user to submit a password somehow. But since HTTPBin (the
primary backend we are using in this tutorial) doesn't do that, you could, as a convenient way to test the demo, add a pre-canned login page to the application, and store it in your Compute program. Start by creating a
login.html page in the
Then add a section to the
main() function to intercept
GET requests to the
/ path and return the login page instead of forwarding the request to origin.
Congratulations! You now have a mechanism to see if the submitted credentials are part of the known compromised credentials.
$ fastly compute publish✓ Initializing...✓ Verifying package manifest...✓ Verifying local rust toolchain...✓ Building package using rust toolchain...✓ Creating package archive...
SUCCESS: Built rust package hibp-enrichment (pkg/hibp-enrichment.tar.gz)
There is no Fastly service associated with this package. To connect to an existing serviceadd the Service ID to the fastly.toml file, otherwise follow the prompts to create aservice now.
Press ^C at any time to quit.
Create new service: [y/N] y
✓ Initializing...✓ Creating service...
Backend (hostname or IP address, or leave blank to stop adding backends): api.pwnedpasswords.comBackend port number:  443Backend name: [backend_1] api
Backend (hostname or IP address, or leave blank to stop adding backends): httpbin.orgBackend port number:  443Backend name: [backend_1] primary
Backend (hostname or IP address, or leave blank to stop adding backends):
✓ Initializing...✓ Creating domain 'random-funky-words.edgecompute.app'...✓ Creating backend 'api' (host: api.pwnedpasswords.com, port: 443)...✓ Creating backend 'primary' (host: httpbin.org, port: 443)...✓ Uploading package...✓ Activating version...
Manage this service at: https://manage.fastly.com/configure/services/PS1Z4isxPaoZGVKVdv0eY
View this service at: https://random-funky-words.edgecompute.app
SUCCESS: Deployed package (service PS1Z4isxPaoZGVKVdv0eY, version 1)
Navigate to the URL shown under "View this service at" in the output above, and you should see the login page. When you log in, your request will be forwarded to HTTPBin, which simply echos back to you what it received. This enables you to see that the origin server has received an additional header with the credential in it.
Try it with no password, with the password '123456', and with something strong and random. You should be able to trigger all three of the possible values of
Now that you understand how to use an API request to enrich data that is sent to the origin, you could combine this with other Fastly sources such as proxy description to gain visibility into if a given client is coming from a proxy. You could also add API requests to other 3rd party sources or your own sources, and perform them in parallel.