You appear to be offline. Some site functionality may not work.

Beacon termination

Your website includes JavaScript on the client side that generates analytics, and you want to collect this data, but want to avoid an uncacheable request reaching your servers for every pageview. Fastly's real time logging can help.

  • LEARN

    Start coding from the instructions below. Make the tests pass!

  • PLAY

    Try out a fully functional example. Just press play.

  • USE

    Add this solution to a service in your Fastly account

Illustration of pattern concept

Beacons are HTTP requests, normally POSTs, sent from a web browser to record some analytics data. When these requests first started to be used on the Web, they would masquerade as 1x1px images, but these days browsers offer native support for beacons via the navigator.sendBeacon method, and native apps will also now often send beacon data back to base.

Data collected from beacons has myriad uses: business metrics, customer behaviour modelling, performance monitoring, and so on. Sometimes you want to see it in real time, and for other use cases you can more efficiently analyse it in batches at the end of the day. Some data benefits from being treated as discrete events, other data looks more like time series. In each case the frequency of delivery and the tool you're going to use to store and process it needs to match your use case.

Fastly supports a wide range of third parties that will ingest your log data. Using these, you can divert beacon request payloads to a log endpoint, and avoid putting load on your own infrastructure.

Instructions

  1. Configure an inert log endpoint

    Start by creating a log endpoint to which you will send your log data. By default, Fastly will log every request, so you need to take care to configure the log endpoint to be inactive - then you can invoke it explicitly from your code.

    Choose from our list of supported logging providers. The logging provider you choose will be a function of how you want to analyse the data. For example, if storing data for later analysis, choose a static object store like Amazon's S3, Google's Cloud Storage, Azure Blob Storage or DigitalOcean Spaces. These generally offer low prices per gigabyte and aren't fussy about how you format your data.

    If you want instant access to the data as it comes in, and want to model your data as discrete events, use a provider like Splunk, Logentries or Papertrail.

    You can even go directly to records in a general purpose database by connecting Google's BigQuery. In that case, you should pay close attention to formatting, since BigQuery is fussy about accepting only JSON formatted data and only a subset of date formats.

    There are two ways to set up a log endpoint:

    To turn off automatic logging, attach an always-false condition to the endpoint. If you're using the UI, the instructions are at the bottom of the guide page we referenced above. In the API, you'll want to look at the methods associated with conditions, create a condition whose statement property is set to "false", and then include the name of the condition when you create (or update) your log endpoint.

    Name your log endpoint something that you can remember; you'll need to reference it from your code later.

  2. Intercept applicable logging requests

    Decide on the path that you will use to receive log data. We can most conveniently process data that comes as a querystring, so to allow for that, don't req.url to match the request path, since it contains the querystring. Instead, you can more conveniently check for your desired path against req.url.path:

    RECV
    if (req.url.path == "/log") {
      error 618 "beacon:log";
    }

    Since you do not intend to send this request to your origin server or use a cached object, you need to treat it as an error. This allows you to transfer control of the request from the recv subroutine to the error subroutine, skipping the cache lookup. We recommend that when triggering a custom error, you should use an error code in the 6xx range, which we guarantee will never be reserved by Fastly, and is also not standardised as part of HTTP. Including an additional custom 'response status text' ('beacon:log' in the example here) further ensures that you will not conflict with any other code you include in your configuration that might use the same error number.

  3. Intercept the error and extract the data to be logged

    You can now create the code in the error subroutine that you need to intercept the error thrown from recv.

    ERROR
    if (obj.status == 618 && obj.response == "beacon:log") {
    
      # ... later steps in the tutorial will add code here
    
    }

    When an error is thrown, an obj variable is created that represents a synthetic HTTP response, and will acquire the status code and response status text defined when it was thrown. It's worth matching both the error status code and the response status text, to ensure that you don't conflict with other errors.

    If your incoming data is on the query string of the URL, then it can already be accessed via req.url.qs.

    If you use navigator.sendBeacon in your frontend JavaScript to create the request, it will have been sent as a POST request, but can still carry the data on the query string, and that's the easiest way to make it available to work with in VCL:

    JavaScript
    let payload = new URLSearchParams();
    payload.append('loadTime', 43.2);
    navigator.sendBeacon('/log?' + payload.toString());

    However, navigator.sendBeacon can also send a payload in the body of the request, and the format it sends it in will depend on which type of object you pass to the sendBeacon function:

    • sendBeacon(string, URLSearchParams): If the payload is a URLSearchParams then the content type will be application/x-www-form-urlencoded (except in Chrome, due to a bug, it will have a Content-type: text/plain but the payload will still be URL encoded form data, ie a query string).
    • sendBeacon(string, FormData): If the payload is a FormData then the content type will be multipart/form-data (this is not recommended, as no parser exists in VCL).
    • sendBeacon(string, string): If the payload is a plain string (such as the output of JSON.stringify) then the content type will be text/plain.
    • sendBeacon(string, Blob): If the payload is a Blob then the content type will be the type specified in the Blob.

    All in all, and especially considering the bug in Chrome, we recommend constructing a URLSearchParams and simply serialising it on the end of the beacon URL as shown above

    Dealing with JSON input

    If not a query string (URLSearchParams), the next most likely format you might want to use for your beacon is JSON. If you are dealing with JSON beacons, you might find it easier just to log the data directly from req.body.

    Both querystrings and request body content are subject to limits on the maximum length we can process. See our resource limits documentation for details.

  4. Enrich the data (optional)

    The overall goal of this solution is to log the data you're receiving from the client (browser or native app), but Fastly has a wealth of additional data about the request that you might want to capture as well. Take a look at our VCL variables to understand what is available. Here we'll show some examples of the most popular data that people tend to like to capture at this point:

    ERROR
      set req.url = querystring.add(req.url, "clientCountry", client.geo.country_code);
      set req.url = querystring.add(req.url, "timestamp", strftime({"%Y-%m-%dT%H:%M:%SZ"}, time.start));
      set req.url = querystring.add(req.url, "serverDatacenter", server.datacenter);
      set req.url = querystring.add(req.url, "autoSystemNumber", client.as.number);
      set req.url = querystring.add(req.url, "clientIP", req.http.fastly-client-ip);
    

    With query string data, you can take advantage of our suite of query manipulation functions to add extra params to the query. In this case, querystring.add is used to add geolocation and IP information about the client.

    Alternative: Enriching JSON

    If you are dealing with incoming JSON in req.body, then you can splice the client-supplied data, and the Fastly variables, into a string variable:

    ERROR
      declare local var.jsonData STRING;
      set var.jsonData = regsub(req.body, "\}$", ", ") +
        {""timestamp": ""} + strftime({"%Y-%m-%dT%H:%M:%SZ"}, time.start) + {"", "}
        {""clientCountry": ""} + client.geo.country_code + {"", "} +
        {""serverDatacenter": ""} + server.datacenter + {"", "} +
        {""autoSystemNumber": "} + client.as.number + {", "} +
        {""clientIP": ""} + req.http.fastly-client-ip + {"""}
      {"}"};

    Here you are using regsub to remove the final } from the incoming JSON, and replacing it with a comma, to allow the addition of more properties. Since JSON syntax makes use of double-quotes, and VCL strings are double-quote delimited, use long string delimiters ({"..."}), which support literal double-quote characters within strings.

  5. Dispatch the data to your log endpoint

    Using the name you chose for your log endpoint earlier, form a log statement to send the data:

    ERROR
      log "syslog " req.service_id " logger_name :: " req.url.qs;

    In place of req.url.qs, substitute req.body if you want to pass through the unmodified POST data, or var.jsonData if you opted to construct a custom JSON object in the previous step.

    Whichever format you use to serialise the log data, be aware that each log event must be on one line, so cannot contain any newline characters. Log events containing newline characters will be truncated at the end of the first line.

  6. Construct a response to the client

    Even though we don't really care about the response, and the browser will ignore it anyway, we should construct a response that's somewhat more appropriate that simply allowing an HTTP 618 beacon:log response to go to the browser, because technically that's invalid HTTP.

    ERROR
      set obj.status = 204;
      set obj.response = "No content";
      set obj.http.cache-control = "no-store, private";

    The most appropriate HTTP response status code is 204, meaning "No content", and for compatibility with HTTP/1.1, you should also set the response status text (from HTTP/2 onwards, the response status text is no longer included in the response).

    It's also important, in case your client consults a cache before making the beacon request, to ensure that the response is not cacheable. If you are using navigator.sendBeacon, the browser will not consult the cache, but it makes sense to ensure that nothing caches this response in any case.

  7. Deliver the response

    The final step within the error subroutine is to tell Fastly to terminate processing of error and move to the deliver stage.

    ERROR
      return (deliver);

    It's a good idea to do this because otherwise subsequent code in the error subroutine might further modify the response obj.

    Your complete error subroutine should now look like this (assuming you are working with querystrings):

    ERROR
    if (obj.status == 618 && obj.response == "beacon:log") {
    
      if (std.strlen(req.body) > 0 && req.method == "POST" && req.http.Content-Type ~ "^application\/x-www-form-urlencoded") {
        set req.url = req.url + if (req.url ~ "\?", "&", "?") req.body;
      }
    
      set req.url = querystring.add(req.url, "clientCountry", client.geo.country_code);
      set req.url = querystring.add(req.url, "serverDatacenter", server.datacenter);
      set req.url = querystring.add(req.url, "autoSystemNumber", client.as.number);
      set req.url = querystring.add(req.url, "clientIP", client.ip);
    
      log "syslog " req.service_id " logger_name :: " req.url.qs;
    
      set obj.status = 204;
      set obj.response = "No content";
      set obj.http.cache-control = "no-store, private";
    
      return (deliver);
    }
  8. Check your log collector

    Once you have activated the solution in your service, send some requests to the log path and ensure that you receive a 204 response.

    The data should now be flowing to your chosen third party logging provider, but if not, log into your service in the Fastly management UI to see if any logging errors are being reported. If we are unable to send data to your log endpoints, the error is shown at the top of the configuration page for your service, or alternatively, load the logging status API endpoint to see the status directly:

    curl "https://api.fastly.com/service/{SERVICE_ID}/logging_status" -H "Fastly-Key: {FASTLY_API_KEY}"

    In particular, note the LastError property for information on why we're unable to send log data to your endpoint. Common reasons for failure of log endpoints to receive data include:

    • Your logging provider is experiencing technical problems
    • The data is not valid and is being rejected by the logging provider's injestion system
    • Your service didn't emit any log events

Next steps

Regardless of how you are receiving the input in this solution, we are not doing any validation. Consider copying valid properties individually from the input to an output variable, which will allow you to both ensure that the formatting is valid, and also to ensure that the value is within allowed bounds. You can also thus discard any excess data that is included in the input but not recognised as a known field name.

See also

Reference docs:

Blog posts:

Quick install

This solution can be added directly to an existing service in a Fastly account as a set of VCL snippets. The embedded fiddle below shows the complete solution. Feel free to run it, and click the 'INSTALL' tab to customise and upload it to your service:

Click to view the fiddle code

Once you have the code in your service, you can further customise it to your needs, but if you keep it unmodified, it will be eligible for automatic upgrades if this recommended solution is improved in future.

All code provided through the Fastly Developer Library is provided under both the BSD and MIT open source licenses.

Get in touch

Help us make this resource more useful for the entire Fastly community. Email your questions, requests, and big ideas to developers@fastly.com — or reach out and let us know what you’re working on.