--- title: 'Log streaming: Amazon S3' summary: null url: >- https://www.fastly.com/documentation/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-amazon-s3 --- Fastly's [Real-Time Log Streaming](https://www.fastly.com/documentation/guides/integrations/streaming-logs/about-fastlys-realtime-log-streaming-features) feature can send log files to [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3). Amazon S3 is a static file storage service used by developers and IT teams. You can also use the instructions in this guide to configure log streaming to another S3-compatible service. > **NOTE:** > > Fastly does not provide direct support for third-party services. Read [Fastly's Terms of Service](https://www.fastly.com/terms) for more information. > > ## Prerequisites Before adding Amazon S3 as a logging endpoint for Fastly services, we recommend creating Identity and Access Management (IAM) credentials in your AWS account specifically for Fastly. Our recommended way for doing this is by creating an AWS IAM role, which lets you grant temporary credentials. For more information, see [Creating an AWS IAM Role for Fastly Logging](https://www.fastly.com/documentation/guides/integrations/streaming-logs/creating-an-aws-iam-role-for-fastly-logging). Alternatively, create an IAM user and grant the user `s3:PutObject` and `s3:AbortMultipartUpload` permissions for the logging stream. For more information, see Amazon's guidance on [understanding and getting your AWS credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). ## Adding Amazon S3 as a logging endpoint After you've registered for an Amazon S3 account and created an IAM user in Amazon S3, follow these instructions to add Amazon S3 as a logging endpoint: ### Cdn Services 1. Review the information in our guide to [setting up remote log streaming](/guides/integrations/streaming-logs/setting-up-remote-log-streaming). 2. In the Amazon Web Services S3 area, click **Create endpoint**. 3. Fill out the **Create an Amazon S3 endpoint** fields as follows: - In the **Name** field, enter a human-readable name for the endpoint. - In the **Placement** area, select where the logging call should be placed in the generated VCL. Valid values are **Format Version Default** and **None**. Read our guide on [changing log placement](/guides/integrations/streaming-logs/changing-log-placement) for more information. - In the **Log format** field, optionally enter an Apache-style string or VCL variables to use for log formatting. Consult the [example format section](#example-format) for details. - *(Optional)* In the **Timestamp format** field, enter a timestamp format for log files. The default is an `strftime` compatible string. Our guide on [changing where log files are written](/guides/integrations/streaming-logs/changing-where-log-files-are-written) provides more information. - In the **Bucket name** field, enter the name of the Amazon S3 bucket in which to store the logs. - _(Optional)_ In the **Domain** field, enter the domain of the Amazon S3 endpoint. If your Amazon S3 bucket was not created in the US Standard region, you must set the domain to match the appropriate endpoint URL. Use the table in the [S3 section of the Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) Amazon S3 documentation page. To use an S3-compatible storage system (such as DreamHost's [DreamObjects](https://www.dreamhost.com/cloud/storage/)), set the domain to match the domain name for that service (for example, in the case of DreamObjects, the domain name would be `objects.dreamhost.com`). - In the **Access method** field, select either **User Credentials** or **IAM Role**. - If you select **User Credentials**, enter the access key and secret key associated with the IAM user you created in your AWS account specifically for Fastly. Check out Amazon's documentation on [security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) for more information. > **HINT:** Password management software may mistakenly treat the **Secret Key** field as a password field because of the way your web browser works. As such, that software may try to auto-fill this field with your Fastly account password. If this happens to you, the AWS integration with Fastly services won't work and you will need to enter **Secret Key** manually instead. - If you select **IAM Role**, enter the Amazon Resource Name (ARN) for the IAM role granting Fastly access to S3. For more information, see [Creating an AWS IAM Role for Fastly Logging](https://www.fastly.com/documentation/guides/integrations/streaming-logs/creating-an-aws-iam-role-for-fastly-logging). - *(Optional)* In the **Period** field, enter an interval (in seconds) to control how frequently your log files are rotated. Rotation entails the finalization of one file object and the start of a new one, never removing any previously created file object. This value defaults to `3600` seconds. - *(Optional)* From the **Processing region** menu, select a geographic region where logs are processed before being sent to the logging endpoint. Our guide on [regional log aggregation](/guides/integrations/streaming-logs/setting-up-regional-log-aggregation) provides more information. 4. Click **Advanced options** and fill out the fields as follows: - *(Optional)* In the **Path** field, enter the path within the bucket to store the files. The path ends with a trailing slash. If this field is left empty, the files will be saved in the bucket's root path. Our guide on [changing where log files are written](/guides/integrations/streaming-logs/changing-where-log-files-are-written) provides more information. - *(Optional)* In the **PGP public key** field, enter a PGP public key that Fastly will use to encrypt your log files before writing them to disk. You will only be able to read the contents by decrypting them with your private key. The PGP key should be in [PEM (Privacy-Enhanced Mail) format](https://en.wikipedia.org/wiki/Privacy-enhanced_Electronic_Mail). Read our guide on [log encryption](/guides/integrations/streaming-logs/encrypting-logs) for more information. - In the **Select a log line format** area, select the log line format for your log messages. Our guide on [changing log line formats](/guides/integrations/streaming-logs/changing-log-line-formats) provides more information. - *(Optional)* In the **Compression** field, select the compression format you want applied to the log files. Our guide on [changing log compression options](/guides/integrations/streaming-logs/changing-log-compression-options) provides more information. - From the **Redundancy level** menu, select a setting. This value defaults to **Standard**. Amazon's [Using Reduced Redundancy Storage Guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) provides more information on using reduced redundancy storage. - _(Optional)_ From the **ACL** menu, select an access control header. See Amazon's [Access Control List (ACL) Specific Request Headers](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html) for more information. - _(Optional)_ **Server side encryption** area, select an encryption method to protect files that Fastly writes to your Amazon S3 bucket. Valid values are **None**, **AES-256**, and **AWS Key Management Service**. If you select **AWS Key Management Service**, you'll have to provide an AWS KMS Key ID. See Amazon's guide on [protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) for more information. - _(Optional)_ In the **Maximum bytes** field, enter the maximum file size in bytes. This value caps the file size and overrides any configured period if the size threshold is exceeded prior to the end of the period. If set to 0, no maximum is set and the file size will not be capped. > **NOTE:** The minimum file size is 1048576 bytes (1 MiB). 5. Click **Create** to create the new logging endpoint. 6. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes. > **NOTE:** Although Fastly continuously streams logs into Amazon S3, the Amazon S3 website and API do not make files available for access until after their upload is complete. ### Example format The following is an example format string for sending data to Amazon S3. Our discussion of [format strings](https://www.fastly.com/documentation/guides/integrations/streaming-logs/custom-log-formats) provides more information. ```plaintext { "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V", "client_ip": "%{req.http.Fastly-Client-IP}V", "geo_country": "%{client.geo.country_name}V", "geo_city": "%{client.geo.city}V", "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V", "url": "%{json.escape(req.url)}V", "request_method": "%{json.escape(req.method)}V", "request_protocol": "%{json.escape(req.proto)}V", "request_referer": "%{json.escape(req.http.referer)}V", "request_user_agent": "%{json.escape(req.http.User-Agent)}V", "response_state": "%{json.escape(fastly_info.state)}V", "response_status": %{resp.status}V, "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V, "response_body_size": %{resp.body_bytes_written}V, "fastly_server": "%{json.escape(server.identity)}V", "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V } ``` ### Compute Services 1. Review the information in our guide to [setting up remote log streaming for Compute](/guides/integrations/streaming-logs/setting-up-remote-log-streaming-for-compute). Additionally, our developer documentation provides more [information about logging](/guides/integrations/non-fastly-services/developer-guide-logging/) with Compute code written in our [supported languages](/reference/compute/sdks/). 2. In the Amazon Web Services S3 area, click **Create endpoint**. 3. Fill out the **Create an Amazon S3 endpoint** fields as follows: - In the **Name** field, enter the endpoint name you specified in your Compute code. For example, in our [Rust code example](/guides/compute/developer-guides/rust/#logging), the name is `my_endpoint_name`. - *(Optional)* In the **Timestamp format** field, enter a timestamp format for log files. The default is an `strftime` compatible string. Our guide on [changing where log files are written](/guides/integrations/streaming-logs/changing-where-log-files-are-written) provides more information. - In the **Bucket name** field, enter the name of the Amazon S3 bucket in which to store the logs. - _(Optional)_ In the **Domain** field, enter the domain of the Amazon S3 endpoint. If your Amazon S3 bucket was not created in the US Standard region, you must set the domain to match the appropriate endpoint URL. Use the table in the [S3 section of the Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) Amazon S3 documentation page. To use an S3-compatible storage system (such as DreamHost's [DreamObjects](https://www.dreamhost.com/cloud/storage/)), set the domain to match the domain name for that service (for example, in the case of DreamObjects, the domain name would be `objects.dreamhost.com`). - In the **Access method** field, select either **User Credentials** or **IAM Role**. - If you select **User Credentials**, enter the access key and secret key associated with the IAM user you created in your AWS account specifically for Fastly. Check out Amazon's documentation on [security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) for more information. > **HINT:** Password management software may mistakenly treat the **Secret Key** field as a password field because of the way your web browser works. As such, that software may try to auto-fill this field with your Fastly account password. If this happens to you, the AWS integration with Fastly services won't work and you will need to enter **Secret Key** manually instead. - If you select **IAM Role**, enter the Amazon Resource Name (ARN) for the IAM role granting Fastly access to S3. For more information, check out [Creating an AWS IAM Role for Fastly Logging](https://www.fastly.com/documentation/guides/integrations/streaming-logs/creating-an-aws-iam-role-for-fastly-logging). - *(Optional)* In the **Period** field, enter an interval (in seconds) to control how frequently your log files are rotated. Rotation entails the finalization of one file object and the start of a new one, never removing any previously created file object. This value defaults to `3600` seconds. 4. Click **Advanced options** and fill out the fields as follows: - *(Optional)* In the **Path** field, enter the path within the bucket to store the files. The path ends with a trailing slash. If this field is left empty, the files will be saved in the bucket's root path. Our guide on [changing where log files are written](/guides/integrations/streaming-logs/changing-where-log-files-are-written) provides more information. - *(Optional)* In the **PGP public key** field, enter a PGP public key that Fastly will use to encrypt your log files before writing them to disk. You will only be able to read the contents by decrypting them with your private key. The PGP key should be in [PEM (Privacy-Enhanced Mail) format](https://en.wikipedia.org/wiki/Privacy-enhanced_Electronic_Mail). Read our guide on [log encryption](/guides/integrations/streaming-logs/encrypting-logs) for more information. - In the **Select a log line format** area, select the log line format for your log messages. Our guide on [changing log line formats](/guides/integrations/streaming-logs/changing-log-line-formats) provides more information. - *(Optional)* In the **Compression** field, select the compression format you want applied to the log files. Our guide on [changing log compression options](/guides/integrations/streaming-logs/changing-log-compression-options) provides more information. - From the **Redundancy level** menu, select a setting. This value defaults to **Standard**. Amazon's [Using Reduced Redundancy Storage Guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) provides more information on using reduced redundancy storage. - _(Optional)_ From the **ACL** menu, select an access control header. See Amazon's [Access Control List (ACL) Specific Request Headers](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html) for more information. - _(Optional)_ In the **Server side encryption** area, select an encryption method to protect files that Fastly writes to your Amazon S3 bucket. Valid values are **None**, **AES-256**, and **AWS Key Management Service**. If you select **AWS Key Management Service**, you'll have to provide an AWS KMS Key ID. See Amazon's guide on [protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) for more information. - _(Optional)_ In the **Maximum bytes** field, enter the maximum file size in bytes. This value caps the file size and overrides any configured period if the size threshold is exceeded prior to the end of the period. If set to 0, no maximum is set and the file size will not be capped. > **NOTE:** The minimum file size is 1048576 bytes (1 MiB). 5. Click **Create** to create the new logging endpoint. 6. From the **Activate** menu, select **Activate on Production** to deploy your configuration changes. > **NOTE:** Although Fastly continuously streams logs into Amazon S3, the Amazon S3 website and API do not make files available for access until after their upload is complete. ### Recommended log format Log messages can take on any format you choose as long as they can be processed from Amazon S3. ## Related content - [Using Amazon S3 as an origin](https://www.fastly.com/documentation/guides/integrations/non-fastly-services/amazon-s3) - [API reference: Amazon S3 log streaming ](https://www.fastly.com/documentation/reference/api/logging/s3/) - [CLI reference: Amazon S3 log streaming](https://www.fastly.com/documentation/reference/cli/logging/s3/)