--- title: Custom log formats summary: null url: >- https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats --- Fastly provides two versions of custom log formats for use when you [set up remote log streaming](https://www.fastly.com/documentation/guides/integrations/streaming-logs/setting-up-remote-log-streaming). All new logging endpoints use the [version 2 custom log format](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats#version-2-log-format) by default. You can [upgrade](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats#upgrading-endpoints-to-use-version-2-log-format) version 1 logging endpoints to the version 2 custom log format. You can also [make version 2 look like version 1](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats#making-version-2-logs-look-like-version-1) for the sake of continuity. We've described the [key advantages](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats#advantages-of-using-the-version-2-custom-log-format) of the version 2 custom log format below. ## Version 2 log format This table details version 2 of Fastly's custom log formats. All variables should be prefixed by a percent sign (`%`), as indicated in the table. | Format String | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `%%` | The percent sign. | | `%a` | The client IP address of the request. Equivalent to [`req.http.Fastly-Client-IP`](https://www.fastly.com/documentation/reference/http/http-headers/Fastly-Client-IP/). | | `%A` | The local IP address. Equivalent to [`server-ip`](https://www.fastly.com/documentation/reference/vcl/variables/server/server-ip/). | | `%B` | The size of response in bytes, excluding HTTP headers. Equivalent to [`resp.body_bytes_written`](https://www.fastly.com/documentation/reference/vcl/variables/client-response/resp-body-bytes-written/). | | `%b` | The size of response in bytes, excluding HTTP headers. In [Common Log Format](https://httpd.apache.org/docs/trunk/logs.html#common) (CLF), that means a `"-"` rather than a `0` when no bytes are sent. | | `%{Foobar}C` | The contents of cookie `Foobar` in the request sent to the server. | | `%D` | The time taken to serve the request, in microseconds. Equivalent to [`time.elapsed.usec`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/time-elapsed-usec/). | | `%{FOOBAR}e` | Not supported. Always returns `"-"`. | | `%f` | The filename. Equivalent to [`req.url`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-url/) with any query string removed, because Varnish has no notion of a file being served. | | `%h` | The remote IP address. Equivalent to [`req.http.Fastly-Client-IP`](https://www.fastly.com/documentation/reference/http/http-headers/Fastly-Client-IP/). | | `%H` | The request protocol. Equivalent to [`req.proto`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-proto/). | | `%{Foobar}i` | The contents of `Foobar:` header lines in the request sent to the server. | | `%I` | Bytes received, including request and headers. Cannot be zero Equivalent to [`req.bytes_read`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-bytes-read/). | | `%k` | The number of keepalive requests handled on this connection. Always returns `0`. | | `%l` | Not supported. Always returns `"-"`. | | `%m` | The request method. Equivalent to [`req.request`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-request/). | | `%{Foobar}n` | Not supported. Always returns `"-"`. | | `%{Foobar}o` | The contents of `Foobar:` header lines in the reply. | | `%O` | Bytes sent, including headers. Cannot be zero. Equivalent to [`resp.bytes_written`](https://www.fastly.com/documentation/reference/vcl/variables/client-response/resp-bytes-written/). | | `%p` | The canonical port of the server serving the request. Always returns `80`. Equivalent to [`server.port`](https://www.fastly.com/documentation/reference/vcl/variables/server/server-port/). | | `%{format}p` | The canonical port of the server serving the request. Valid formats are `canonical`, `local`, or `remote`. Returns `80` for HTTP requests and `443` for HTTPS requests. | | `%P` | Not supported. Always returns `"-"`. | | `%{format}P` | Not supported. Always returns `"-"`. | | `%q` | The query string (prepended with a `?` if a query string exists, otherwise an empty string). Equivalent to [`req.url`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-url/). | | `%r` | The first line of the request. | | `%R` | Not supported. Always returns `"-"`. | | `%s` | The status. For requests that got internally redirected, this is the status of the _original_ request. Use `%>s` for the final status. Equivalent to [`resp.status`](https://www.fastly.com/documentation/reference/vcl/variables/client-response/resp-status/). | | `%t` | The time the request was received, in Standard English format (e.g., `01/Jan/1970:00:00:00 -0700`). The last number indicates the time zone offset from GMT. | | `%{format}t` | The time, in the form given by `format`, which should be in `strftime(3)` format (potentially localized). If the format starts with `begin:` (the default) the time is taken at the beginning of the request processing. If it starts with `end:` it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by `strftime(3)`, the following format tokens are supported: `sec` (number of seconds since the Epoch), `msec` (number of milliseconds since the Epoch), `usec` (number of microseconds since the Epoch), `msec_frac` (millisecond fraction), and `usec_frac` (microsecond fraction). Equivalent to [`time.start`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/time-start/). | | `%T` | The time taken to serve the request, in seconds. Equivalent to [`time.elapsed.sec`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/time-elapsed-sec/). | | `%u` | Not supported. Always returns `"-"`. | | `%U` | The URL path requested, not including any query string. Equivalent to [`req.url.path`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-url-path/). | | `%v` | The domain name of the request. Equivalent to [`req.http.host`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-http/). | | `%V` | The same as `%v`. Equivalent to [`req.http.host`](https://www.fastly.com/documentation/reference/vcl/variables/client-request/req-http/). | | `%{vcl}V` | The literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., `%{client.geo.country_code}V` or `%{tls.client.cipher}V`). This %-directive is a Fastly extension and is not found in Apache. | | `%X` | The connection status when response is completed. Always set as `+` (connection may be kept alive after the response is sent). | ## Version 1 log format This table details version 1 of Fastly's custom log formats. All variables should be prefixed by a percent sign (`%`), as indicated in the table. | Format String | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `%b` | The content size of the response, calculated using the `Content-Length` header rather than actually checking the length of the response (and may therefore be wrong). | | `%h` | The remote IP address. | | `%l` | The remote log name. Always returns the hardcoded value `"-"`. | | `%r` | The HTTP verb and request path (e.g., `GET /index.html`). Unlike Apache and version 2 log formats, the protocol version is not included. | | `%>s` | The status of the last request. | | `%t` | The time the request was received, in Unix `ctime` format (e.g., `Thu, 01 Jan 1970 00:00:00 GMT`) rather than Apache's Standard English format (e.g., `01/Jan/1970:00:00:00 -0700`). | | `%u` | Always returns `"-"`. | ## Upgrading endpoints to use version 2 log format > **WARNING:** Upgrading is a permanent change. Logging objects using [version 2 formatting](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats#version-2-log-format) cannot be downgraded to version 1. Follow these instructions to upgrade a logging endpoint to the version 2 custom log format: 1. Log in to the [Fastly control panel](https://manage.fastly.com). 2. From the [**Home**](https://manage.fastly.com/home) page, select the appropriate service. You can use the search box to search by ID, name, or domain. 3. Click **Edit configuration** and then select the option to clone the active version. 4. Click **Logging**. ![the logging endpoints page with upgrade message](/img/upgrade-logging-endpoints.png) 5. Click the name of a logging endpoint you want to edit. ![the edit logging endpoint page with upgrade message](/img/edit-logging-endpoint-upgrade.png) 6. Click **Convert to Log Format Version 2**. ![the convert to log format version 2 window](/img/convert-log-format-v2.png) 7. Select an output format: - **Use compatible output** is the recommended setting. This setting won't modify your timestamp format string, but your logs will be formatted differently. The new format will be compatible with Apache's log format. - **Maintain legacy output** uses the version 2 parser, but the generated log string will be the same. This means that any instances of `%t` need to be turned into `%{now}V`, any instances of `%r` need to be turned into `%{req.url}V`, and any instances of `%b` need to be turned into `%{resp.http.Content-Length}V`. 8. Click **Select**. 9. Click **Update** to upgrade the logging endpoint to the version 2 custom log format. 10. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes. ### Using the API to upgrade To upgrade a logging endpoint using the [Fastly API](https://www.fastly.com/documentation/reference/api/), either clone the active version of the service you need upgraded or choose a version of the service that is unlocked and not active, then run the following command on that in development version: ```term copy $ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service//version//logging//' --data-binary '{"format_version":"2"}' ``` Keep in mind that the `format_version` field is a per-object field. Updating it on one logging object will _not_ change it on any other objects. For example, to update a Google Cloud Storage logging endpoint named "GCS Test," the curl command would look something like this: ```term copy $ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/1/logging/gcs/GCS%20Test' --data-binary '{"format_version":"2"}' ``` > **HINT:** The `log name` included a space that needed to be [URL encoded](https://en.wikipedia.org/wiki/Percent-encoding) (the space into `%20`). ## Determining which logging version is being used To determine which logging version your service currently uses, issue the following curl command: ```term copy $ curl -X GET -H 'Fastly-Key: FASTLY_API_TOKEN' 'https://api.fastly.com/service//version//logging//' ``` The curl command will produce JSON output detailing the configuration of your service version. For example: ```json { "address": "logs.papertrailapp.com", "created_at": "2016-04-01T15:37:30+00:00", "deleted_at": null, "format": "time.start.msec time.to_first_byte time.elapsed req.body_bytes_read req.bytes_read resp.http.content-length server.region client.ip %>s \"req.method req.url req.proto\" \"req.http.referer\" \"req.http.user-agent\"", "format_version": "2", "hostname": "logs.papertrailapp.com", "name": "fastly", "port": "11111", "public_key": null, "response_condition": "LOG /", "service_id": "1a2b3c4d5e6f7g8h9j0k", "updated_at": "2016-04-01T19:47:47+00:00", "version": "123" } ``` The `format_version` field displays either a `1` or a `2` as appropriate for the custom log format being used. ## Advantages of using the version 2 custom log format The key advantages of using the version 2 custom log format include the following: - Log lines are generated in `vcl_log` instead of `vcl_deliver` to allow us to accurately set the various size variables because `vcl_log` is run after the object has been delivered to the browser. - The `%t` time directive is compatible with Apache log format. In version 1, we used a non-standard time format. - The `%r` "first line of request" directive is compatible with Apache log format. In version 1, we incorrectly left off the protocol. - When using the `%b` directive, which represents the size of a response in bytes, excluding HTTP headers is more accurate. In version 1, we used the reported Content-Length from the origin, which could be inaccurate (especially with [ESI](https://www.fastly.com/documentation/reference/vcl/statements/esi/)). - We've added all Apache logging directives that make sense. In version 1, we used a smaller subset. ## Making version 2 logs look like version 1 The default logging format for version 1 is as follows: ```text %h %l %u %t %r %>s ``` Most of the directives in version 2 are exactly the same - only `%t` and `%r` are different. After you upgrade to version 2 log formats, you can recreate the appearance of version 1 logs using the new `%{...}V` directive, which allows you to specifically include VCL in logging directives: ```text %h %l %u %{now}V %{req.method}V %{req.url}V %>s ``` In addition, if you are using the `%b` directive in version 1, then you can use this directive instead: ```text %{resp.http.Content-Length}V ``` ## Related content - [About Fastly's Real-Time Log Streaming features](https://www.fastly.com/documentation/guides/integrations/streaming-logs/about-fastlys-realtime-log-streaming-features) - [Useful variables to log](https://www.fastly.com/documentation/guides/integrations/streaming-logs/useful-variables-to-log) - [Logging API documentation](https://www.fastly.com/documentation/reference/api/logging/)