Bucket lifecycle management for Object Storage

Bucket lifecycle management lets you define rules that automatically expire objects in a Fastly Object Storage bucket after a set number of days, without having to delete them yourself. This is useful for cleaning up data on a schedule, such as rotating log files, removing temporary uploads, or enforcing a retention window for objects you only need for a limited time. This guide covers the elements that make up a lifecycle rule, how Fastly evaluates those rules, the S3-compatible API operations used to configure them, and how to apply and manage lifecycle configuration on your buckets.

Because Fastly Object Storage charges for stored data as long as it remains in a bucket, lifecycle rules give you a way to automate removal so you aren't paying for data you no longer need.

Before you begin

Make sure you review the prerequisites for using Fastly Object Storage.

Lifecycle rule reference

A lifecycle rule supports the following elements:

Unsupported elements

The following elements are part of the S3 lifecycle configuration API but are not supported. Requests that include them are rejected:

Limitations and considerations

• Each bucket may have a max of 25 rules.
• The minimum Expiration.Days value is 1 day.
• The ID limit is 255 characters and cannot start with fst- or fastly-.
• Filter.Prefix can be up to 1024 bytes.

Rule evaluation

How Fastly evaluates lifecycle rules depends on how many rules match a given object and when the evaluation runs.

Bucket-wide rules

An empty Filter.Prefix (<Prefix></Prefix>, or a <Filter> with no Prefix element at all) matches every object in the bucket, letting you apply a single expiration rule bucket-wide.

Overlapping rules

If more than one rule matches an object (for example, a bucket-wide rule and a more specific prefix rule both match the same object) the shortest expiration wins.

This holds regardless of which rule's prefix is more specific. For example, given the following two rules, an object at tmp/upload.dat matches both: the bucket-wide rule (empty prefix) and the tmp/ rule. Even though tmp/ is the more specific prefix, it doesn't take precedence. The object expires after 1 day, because the bucket-wide rule's expiration is shorter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<LifecycleConfiguration>
  <Rule>
    <ID>expire-all-objects</ID>
    <Filter>
      <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>1</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>expire-temp-files</ID>
    <Filter>
      <Prefix>tmp/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Evaluation timing

Fastly evaluates lifecycle rules once per day, around midnight UTC. Depending on when an object becomes eligible for expiration relative to that evaluation window, it can take up to 48 hours for the object to actually be removed after it becomes eligible.

S3-compatible operations

Fastly Object Storage exposes lifecycle configuration through three S3-compatible API operations:

GET returns the lifecycle configuration set on the bucket. If no rules are configured, it returns a 404 Not Found with a NoSuchLifecycleConfiguration error code. DELETE removes all lifecycle rules configured on the bucket and returns 204 No Content.

PUT requests must include a Content-MD5 header or an x-amz-checksum-* header (using CRC32, CRC32C, SHA1, or SHA256) for payload integrity. Requests signed with AWS Signature Version 4 (SigV4) using a signed payload satisfy this requirement automatically. The AWS CLI and SDKs handle this for you. An example VCL Snippet can be found on the working with Object Storage page.

Because PUT replaces the full configuration, tools that don't specify an explicit ID for each rule, including some infrastructure-as-code providers like Terraform, will see a new, different ID generated on every apply. To avoid this, always specify an explicit ID for each rule.

Authentication and permissions

Lifecycle operations support both SigV4 and Fastly API token authentication (using the Fastly-Key header).

Managing lifecycle configuration requires an access key with full access to all buckets:

• Retrieving a lifecycle configuration requires the read-only-admin access key permission.
• Setting or removing a lifecycle configuration requires the read-write-admin access key permission.

Access keys with limited access to specific buckets (read-write-objects, read-only-objects) can't manage lifecycle configuration.

Managing bucket lifecycle configuration

To manage lifecycle configuration, save a configuration to a JSON file and use the AWS CLI (or any S3-compatible client) to apply it. For example, the following file expires objects under a logs/ prefix after 90 days:

1
2
3
4
5
6
7
8
9
10

{
  "Rules": [
    {
      "ID": "expire-old-logs",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Expiration": { "Days": 90 }
    }
  ]
}

Apply it to a bucket:

$ aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --profile fastly-us-east-1 --lifecycle-configuration file://lifecycle.json

You can also retrieve the configuration set on a bucket:

$ aws s3api get-bucket-lifecycle-configuration --bucket my-bucket --profile fastly-us-east-1

Or remove all lifecycle rules from a bucket entirely:

$ aws s3api delete-bucket-lifecycle --bucket my-bucket --profile fastly-us-east-1

Example configurations

Expire objects under a logs/ prefix after 90 days:

1
2
3
4
5
6
7
8
9
10
11
12

<LifecycleConfiguration>
  <Rule>
    <ID>expire-old-logs</ID>
    <Filter>
      <Prefix>logs/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>90</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Apply different retention periods to different prefixes in the same bucket:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<LifecycleConfiguration>
  <Rule>
    <ID>expire-temp-files</ID>
    <Filter>
      <Prefix>tmp/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>1</Days>
    </Expiration>
  </Rule>
  <Rule>
    <ID>expire-old-logs</ID>
    <Filter>
      <Prefix>logs/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>90</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Expire every object in the bucket after one year, regardless of prefix:

1
2
3
4
5
6
7
8
9
10
11
12

<LifecycleConfiguration>
  <Rule>
    <ID>expire-all-objects</ID>
    <Filter>
      <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Expiration>
      <Days>365</Days>
    </Expiration>
  </Rule>
</LifecycleConfiguration>

Related content

• Working with Object Storage
• Object Storage Product API