You appear to be offline. Some site functionality may not work.
Call Us

Introducing scripted testing for Fastly fiddle

By  Andrew Betts, Web Developer and Principal Developer Advocate, October 8, 2019 in Testing

Fastly Fiddle allows for instant experimentation with Fastly's edge cloud behaviours without having to set up a Fastly account. Now, we're adding the ability to define assertions to specify the behaviour you are trying to create.

With the recent launch of our library of solution recipes that make it easier to build on Fastly, we now can demonstrate a selection of the solutions that people have created to run on Fastly.  Our growing library of recipes is nearing a hundred, and in building these, we realised that we needed a better way not only to verify that the solutions continue to work as we evolve our network, but also to make it clear what the solution is achieving.

Test for fiddles

Fiddle presents a lot of information— and it can be overwhelming.  How do you know what the key point is that the solution has actually achieved? We already automatically highlight 'interesting' HTTP headers, based on an analysis of your VCL code, but now, you can explicitly define what your fiddle is supposed to do:

Test results in fiddle screenshot

To define tests, we start by writing a fiddle containing the logic that we want to test. Taking a simple example, we could add a header to each request as it comes into Fastly, so that the information in that new header is transmitted to your origin.

Here's a fiddle that does that. Feel free to run it:

Testing this behaviour in a normal Fastly service by external observation would be hard. There's no evidence in the response back to the browser that your header got added to the request to the origin. What we need to do is be able to assert things about the internal behaviour of Fastly.

Test targets: what we can test

There are three major sources of data about requests transiting Fastly: the client-side request/response, the origin request/response, and the events that are triggered within the Fastly platform. We expose these in our test syntax as clientFetch, originFetches (there may be more than one), and events.

These three top level objects expose lots of interesting and useful properties:

clientFetch Obj The request from the client to Fastly (and the response from Fastly)
.req Str HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body
.resp Str HTTP response header, contains response status line and response headers (not the body)
.respType Str Parsed Content-type response header value (mime type only)
.isText Bool Whether the response body can be treated as text
.isImage Bool Whether the response body can be treated as an image
.status Num HTTP response status
.bodyPreview Str UTF-8 text preview of the body (truncated at 1K)
.bodyBytesReceived Num Amount of data received
.bodyChunkCount Num Number of chunks received
.complete Bool Whether the response is complete
.trailers Str HTTP response trailers
originFetches Array Origin fetches made during the request
[idx] Obj Each fetch is one object
.vclFlowKey Str ID for the VCL flow that triggered this fetch
.req Str HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body
.resp Str HTTP response header, contains response status line and response headers (not the body)
.remoteAddr Str Resolved IP address of origin
.remotePort Num Port on origin server
.remoteHost Str Hostname of origin server
.elapsedTime Num Total time spent on origin fetch (ms)
events Array Fastly VCL events related to the request, in VCL flow order. May contain multiple VCL flows if there are restarts, ESIs or shielding.
[idx] Obj Each array element is one VCL event
.fnName Str The type of the event. May be 'recv', 'hash', 'hit', 'miss', 'pass', 'waf', 'fetch', 'deliver', 'error', or 'log'
.datacenter Str Three letter code identifying the Fastly data center location in which this event occurred, eg. 'LCY', 'JFK', 'SYD'
.nodeID Str Numeric identifier of the individual server on which this event occurred.
.originFetch Obj the origin fetch associated with the event (see originFetches above for the model). Only FETCH events have an originFetch property.
.logs Array Array of strings, messages logged from this event
.<various> Any property reported in the fiddle UI for an event can be targeted for testing. For example: within MISS events, a staleExists property is reported.
logs Array Array of strings, messages logged from all VCL events
insights Array Insight tags for this request. Insight tags identify recommendations or divergences from best practices. For example: [ "client-cc-missing", "invalid-header" ]

To get to the headers that have been added to the origin request, we need to access the req property of the first item in the originRequests collection:

originRequests[0].req

Simple enough, but what if you wanted to also check that the response from the origin was cached? We would need to access the .ttl property of the fetch event, but we can't be sure what the index of the fetch event is within the events collection.  What is needed is a way to filter the events collection for just the fetch events, and then pick the first one.

Aggregation functions

To make accessing test targets easier, we've added a set of aggregation functions. Drop these into the target string to transform the data:

listBy(field) Array => Array

Takes an array of objects and makes an array of arrays, where in each subarray, all objects share the same value of `field`. Items in the output array are ordered based on the order in which the values of the selected field first appear in the input.

events.listBy(vclflowkey)[1][0].fnName is "recv"

where(field=val) Array => Array

Takes an array of objects and filters it to leave only those where the property called `field` has value `val`.

events.where(fnName=recv)[0].url startsWith "/a/"

groupBy(field) Array => Obj

Takes an array of objects and splits it into multiple arrays where each one has the same value for the property `field`, and organises the resulting data into an object where the `field` values are keys.

events.groupBy(fnName).recv[0].url startsWith "/a/"

transpose() Array => Obj

Takes an array of objects and makes an object of arrays.  Where multiple input objects share the same property name, that property becomes a top level property with an array containing all the values.

events.transpose().return notIncludes "error"

count() Array => Num

Takes an array and returns the length

originFetches.count() greaterThan 1

concat() Array => Str

Takes an array and returns a string representation of all array items joined together, delimited by newlines.

logs.concat() includes "Hello"

So, now we can identify the first fetch event's ttl property with:

events.where(fnName=fetch)[0].ttl

Finally, you need to assert something about this target data.

Assertions

We support any assertion defined in Chai, and have added a few more. These are likely to be the ones that are most useful to you:

Name Value type Description
is Any Non-strict equality (using ==)
isJSON None The target is valid JSON (does not require a reference value, so just two parameters)
isTrue None The target is true
isAtLeast, isAbove JSON number Target is numerically higher than the reference value
isAtMost, isBelow JSON number Target is numerically lower than the reference value
includes Any The target includes the reference value. Can be used to assert the inclusion of a value in an array, a substring in a string, or a subset of properties in an object.
matches JS RegExp The target matches a regular expression. Regex must be delimited with / and may be followed by modifiers, eg /abc/i
oneOf JSON array Checks that the value of the target is equal to at least one of the values in the reference array
startsWith JSON string Checks that the value of the target starts with the reference string
endsWith JSON string Checks that the value of the target ends with the reference string

So, now we can form the full tests for our example fiddle:

originFetches[0].req includes "custom-header"
events.where(fnName=fetch)[0].ttl isAtLeast 3600000000

Note that the ttl property, along with all timing properties reported by Fastly, is in microseconds - so 3600 seconds (1 hour) is 3,600,000,000 μs. Let's look at how these tests are added to a fiddle through the UI:

Adding tests in fiddle UI

Try running the fiddle again, but this time watch as the tests are run!

Tests are parsed when your fiddle runs, and will tell you if any of your test syntax is unparseable.

Debugging

This is all well and good if your tests are passing, but what if they are failing? How do you know that you wrote the test correctly, vs the fiddle itself having a bug that the test has caught?

If a test fails when it is run, Fiddle will display not just the failure state but also the actual value of the target:

Test with error

This can also be a great help when constructing the target string. If you're not sure what properties exist on a target path, you can just write a test using that path, allowing it to fail so you can see what the target actually looks like:

Tests will run asynchronously and need to wait for asynchronous instrumentation to be delivered. For some targets, the delivery of the instrumentation data takes longer than others.  This is indicated by the hourglass icon on each test — no hourglass, no delay; running hourglass, short delay; completed hourglass, longer delay.

Detailed debug when tests fail in fiddle

We hope that being able to run tests in fiddle helps to validate Fastly configurations and provide additional confidence in moving your logic to the edge. If you have any ideas or suggestions, feel free to post below in the comments or drop me a line.

Author

Andrew Betts | Web Developer and Principal Developer Advocate

Andrew Betts is a Web Developer and Principal Developer Advocate for Fastly, where he works with developers across the world to help make the web faster, more secure, more reliable, and easier to work with. He founded a web consultancy which was ultimately acquired by the Financial Times, led the team that created the FT’s pioneering HTML5 web app, and founded the FT’s Labs division. He is also an elected member of the W3C Technical Architecture Group, a committee of nine people who guide the development of the World Wide Web.

triblondon