---
title: 'Log streaming: Elasticsearch'
summary: null
url: >-
  https://www.fastly.com/documentation/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-elasticsearch
---

Fastly's [Real-Time Log Streaming](https://www.fastly.com/documentation/guides/integrations/streaming-logs/about-fastlys-realtime-log-streaming-features) feature can send log files to [Elasticsearch](https://www.elastic.co/elastic-stack). Elasticsearch is a distributed, RESTful search and analytics engine.

> **NOTE:** 
>
> Fastly does not provide direct support for third-party services. Read [Fastly's Terms of Service](https://www.fastly.com/terms) for more information.
>
>

## Prerequisites

Before adding Elasticsearch as a logging endpoint for Fastly services, ensure Elasticsearch is running on a remote server. You’ll need to know the endpoint URL that includes a port to which logs should be sent (make sure it can receive traffic from Fastly) and also the name of the index to send logs to. For more information on setting up Elasticsearch, check out the [Elasticsearch setup documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html).

This logging endpoint works with all actively supported versions of Elasticsearch as well as some versions that have already reached their end-of-life. We also work with OpenSearch server integration. Other distributions that are API-compatible with Elasticsearch may also work but have not been explicitly tested and are not guaranteed.

### Required privileges

We send data using the [Bulk API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) via the `index` action. When using basic authentication, ensure that the [required index privileges](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk-1) to use the `index` action are granted to the user role.

We also require access to the root path API of the Elasticsearch server. This API returns metadata about the server, such as the version number, that allows our integration to make the best choice about which bulk data API to use for each customer's server. Access to this API allows us to properly work with the wide range of Elasticsearch versions used by our customers as well as other Elasticsearch-compatible distributions.

## Adding Elasticsearch as a logging endpoint

Follow these instructions to add Elasticsearch as a logging endpoint:

### Cdn Services

1.   Review the information in our guide to [setting up remote log streaming](/guides/integrations/streaming-logs/setting-up-remote-log-streaming).

2. In the Elasticsearch area, click **Create endpoint**.
3. Fill out the **Create an Elasticsearch endpoint** fields as follows:
   -   In the **Name** field, enter a human-readable name for the endpoint.

   -   In the **Placement** area, select where the logging call should be placed in the generated VCL. Valid values are **Format Version Default** and **None**. Read our guide on [changing log placement](/guides/integrations/streaming-logs/changing-log-placement) for more information.

   - In the **Log format** field, enter the data to send to Elasticsearch. See the [example format section](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-elasticsearch#example-format) for details.
   - In the **URL** field, enter the Elasticsearch endpoint URL that includes a port to which logs should be sent. The URL must be sent using HTTPS on a port that can receive incoming TCP traffic from Fastly.
   - In the **Index** field, enter the name of the Elasticsearch index to send logs to. The index must follow the Elasticsearch [index format rules](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html). We support [strftime](https://man7.org/linux/man-pages/man3/strftime.3.html) interpolated variables inside braces prefixed with a pound symbol. For example, `#{%F}` will interpolate as YYYY-MM-DD with today's date.
   - _(Optional)_ In the **Pipeline** field, enter the ID of the Elasticsearch [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) to apply pre-process transformations to before indexing (for example, `my_pipeline_id`).
   - _(Optional)_ In the **Maximum logs** field, enter the maximum number of logs to append to a batch, if non-zero.
   - _(Optional)_ In the **Maximum bytes** field, enter the maximum size of the log batch.
   - _(Optional)_ In the **BasicAuth user** field, enter your [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) username.
   - _(Optional)_ In the **BasicAuth password** field, enter your basic authentication password.
   -   In the **TLS hostname** field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.

   - _(Optional)_ In the **TLS CA certificate** field, copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
   - _(Optional)_ In the **TLS client certificate** field, copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
   - _(Optional)_ In the **TLS client key** field, copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
   -   *(Optional)* From the **Processing region** menu, select a geographic region where logs are processed before being sent to the logging endpoint. Our guide on [regional log aggregation](/guides/integrations/streaming-logs/setting-up-regional-log-aggregation) provides more information.

4.   Click **Create** to create the new logging endpoint.

5.   From the **Activate** menu, select **Activate on Production** to deploy your configuration changes.

### Example format

Data sent to Elasticsearch must be serialized as a JSON object. Here's an example format string for sending data to Elasticsearch:

```plaintext
{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "geo_country": "%{client.geo.country_name}V",
  "geo_city": "%{client.geo.city}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_protocol": "%{json.escape(req.proto)}V",
  "request_referer": "%{json.escape(req.http.referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}
```

### Compute Services

1.   Review the information in our guide to [setting up remote log streaming for Compute](/guides/integrations/streaming-logs/setting-up-remote-log-streaming-for-compute). Additionally, our developer documentation provides more [information about logging](/guides/integrations/non-fastly-services/developer-guide-logging/) with Compute code written in our [supported languages](/reference/compute/sdks/).

2. In the Elasticsearch area, click **Create endpoint**.
3. Fill out the **Create an Elasticsearch endpoint** fields as follows:
   -   In the **Name** field, enter the endpoint name you specified in your Compute code. For example, in our [Rust code example](/guides/compute/developer-guides/rust/#logging), the name is `my_endpoint_name`.

   - In the **URL** field, enter the Elasticsearch endpoint URL that includes a port to which logs should be sent. The URL must be sent using HTTPS on a port that can receive incoming TCP traffic from Fastly.
   - In the **Index** field, enter the name of the Elasticsearch index to send logs to. The index must follow the Elasticsearch [index format rules](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html). We support [strftime](https://man7.org/linux/man-pages/man3/strftime.3.html) interpolated variables inside braces prefixed with a pound symbol. For example, `#{%F}` will interpolate as YYYY-MM-DD with today's date.
   - _(Optional)_ In the **Pipeline** field, enter the ID of the Elasticsearch [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) to apply pre-process transformations to before indexing (for example, `my_pipeline_id`).
   - _(Optional)_ In the **Maximum logs** field, enter the maximum number of logs to append to a batch.
   - _(Optional)_ In the **Maximum bytes** field, enter the maximum size of the log batch.
   - _(Optional)_ In the **BasicAuth user** field, enter your [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) username.
   - _(Optional)_ In the **BasicAuth password** field, enter your basic authentication password.
   -   In the **TLS hostname** field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.

   - _(Optional)_ In the **TLS CA certificate** field, copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
   - _(Optional)_ In the **TLS client certificate** field, copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
   - _(Optional)_ In the **TLS client key** field, copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
4.   Click **Create** to create the new logging endpoint.

5.   From the **Activate** menu, select **Activate on Production** to deploy your configuration changes.

## Related content

- [API reference: Elasticsearch log streaming](https://www.fastly.com/documentation/reference/api/logging/elasticsearch/)
- [CLI reference: Elasticsearch log streaming](https://www.fastly.com/documentation/reference/cli/logging/elasticsearch/)