--- title: 'Log streaming: Hydrolix' summary: null url: >- https://www.fastly.com/documentation/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-hydrolix --- Fastly's [Real-Time Log Streaming](https://www.fastly.com/documentation/guides/integrations/streaming-logs/about-fastlys-realtime-log-streaming-features) feature can send log files to [Hydrolix](https://www.hydrolix.io/), a cloud-based time-series data platform. Hydrolix provides a native integration for Fastly log storage and analysis through Fastly's [HTTPS logging endpoint](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-https). Hydrolix lets you ingest and query those logs in real-time. > **NOTE:** > > Fastly does not provide direct support for third-party services. Read [Fastly's Terms of Service](https://www.fastly.com/terms) for more information. > > ## Prerequisites If you don't already have a Hydrolix account, you'll need to sign up on the [Hydrolix website](https://www.hydrolix.io/). You'll also need to know the following about the target Hydrolix environment: - Instance name - [Project name](https://docs.hydrolix.io/docs/projects-and-tables) - [Table name](https://docs.hydrolix.io/docs/projects-and-tables#tables) > **HINT:** Consider reading [Hydrolix's documentation on integrating with Fastly](https://docs.hydrolix.io/docs/fastly-logs). ## Adding Hydrolix as a logging endpoint > **IMPORTANT:** This logging endpoint is only available for Fastly's Deliver (VCL-based) services, not for Compute services. Follow these instructions to add Hydrolix as a logging endpoint: ### Cdn Services 1. Review the information in our guide to [setting up remote log streaming](/guides/integrations/streaming-logs/setting-up-remote-log-streaming). 2. In the HTTPS area, click **Create endpoint** button. 3. Fill out the **Create an HTTPS endpoint** fields as follows: - In the **Name** field, enter a human-readable name for the endpoint. - In the **Placement** area, select where the logging call should be placed in the generated VCL. Valid values are **Format Version Default** and **None**. Read our guide on [changing log placement](/guides/integrations/streaming-logs/changing-log-placement) for more information. - In the **Log format** field, replace the placeholder log format and make the appropriate changes as shown in our [log format and recommendations](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-hydrolix#log-format-recommendations) section below. - In the **URL** field, enter `https://.hydrolix.live/ingest/event`, replacing `` with the name of your Hydrolix instance. - In the **Maximum logs** field, leave as `0` (the default). - In the **Maximum bytes** field, enter `0`. - *(Optional)* From the **Processing region** menu, select a geographic region where logs are processed before being sent to the logging endpoint. Our guide on [regional log aggregation](/guides/integrations/streaming-logs/setting-up-regional-log-aggregation) provides more information. 4. Click the **Advanced options** link of the **Create an HTTPS endpoint** page. The Advanced options appear. 5. Fill out the **Advanced options** of the **Create an HTTPS endpoint** page as follows: - In the **Content type** field, enter `application/json`. - In the **Custom header name** field, enter `x-hdx-table`. - In the **Custom header value** field, enter `.`, substituting the values for your Hydrolix project and table names. - From the **Method** controls, select **POST**. - From the **JSON log entry format** controls, select **Newline delimited**. - Leave the **Select a log line format** and **Placement** controls set to the defaults. - In the **TLS hostname** field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported. - Leave **TLS CA certificate** field, **TLS client certificate** field and **TLS client key** field all empty. 6. Click **Create** to create the new logging endpoint. 7. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes. ### Log format recommendations For this example we use the log format shown below. You can customize this format with any values you want as long as you also modify your **[Transform](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-hydrolix#creating-a-transform-schema)** and **[View](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-hydrolix#leveraging-views)** configurations. ```plain { "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t", "time_elapsed":%{time.elapsed.usec}V, "is_tls":%{if(req.is_ssl, "true", "false")}V, "client_ip":"%{req.http.Fastly-Client-IP}V", "geo_city":"%{client.geo.city}V", "geo_country_code":"%{client.geo.country_code}V", "request":"%{req.request}V", "host":"%{req.http.Fastly-Orig-Host}V", "url":"%{json.escape(req.url)}V", "request_referer":"%{json.escape(req.http.Referer)}V", "request_user_agent":"%{json.escape(req.http.User-Agent)}V", "request_accept_language":"%{json.escape(req.http.Accept-Language)}V", "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V", "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V" } ``` ## Configuring Hydrolix Streaming Intake Once you have your `project` and `table` setup and Fastly is configured to send logs to your Hydrolix instance, you can focus on setting up the Fastly Log streaming ingest pipeline by defining an [ingest transform schema](https://docs.hydrolix.io/docs/loading-the-basics). ### Creating a transform schema Below is the suggested transform schema to use with the [recommended log format](https://www.fastly.com/documentation/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-hydrolix#log-format-recommendations) described above. Be sure to replace `` with the UUID of your target Hydrolix table. You need to have this transform setup as the default, so `"is_default"` is set to `true`. ```plain { "name": "fastly_transform", "type": "json", "table": "
", "description": "fastly https logs", "settings": { "is_default": true, "output_columns": [ { "name": "timestamp", "datatype": { "type": "datetime", "primary": true, "format": "2006-01-02T15:04:05" } }, { "name": "time_elapsed", "datatype": { "type": "uint64" } }, { "name": "is_tls", "datatype": { "type": "bool" } }, { "name": "client_ip", "datatype": { "type": "string" } }, { "name": "geo_city", "datatype": { "type": "string" } }, { "name": "geo_country_code", "datatype": { "type": "string" } }, { "name": "request", "datatype": { "type": "string" } }, { "name": "host", "datatype": { "type": "string" } }, { "name": "url", "datatype": { "type": "string" } }, { "name": "request_referer", "datatype": { "type": "string" } }, { "name": "request_user_agent", "datatype": { "type": "string" } }, { "name": "request_accept_language", "datatype": { "type": "string" } }, { "name": "request_accept_charset", "datatype": { "type": "string" } }, { "name": "cache_status", "datatype": { "type": "string" } } ] } } ``` Once you define the transform schema, Hydrolix is configured to accept the incoming Fastly log data. ### Leveraging views Hydrolix supports having many different query formats for a single dataset. The query data structure, or view, for a given dataset allows you to customized the queried data and restrict a user’s access to a specific set of columns. Hydrolix automatically generates a default view upon transform creation that can be used to immediately query the dataset - no additional configuration is required. However, you are encouraged to familiarize yourself with the view concept and the benefits that the feature can provide. More detailed information can be found on the [Hydrolix site](https://docs.hydrolix.io/docs/views-1). ## Further reading Hydrolix provides a [tutorial](https://docs.hydrolix.io/docs/fastly-logs) for querying and analyzing Fastly data from within their system. ## Related content - [API reference: HTTPS log streaming](https://www.fastly.com/documentation/reference/api/logging/https/) - [CLI reference: HTTPS log streaming](https://www.fastly.com/documentation/reference/cli/logging/https/)