On-Demand Migration for Fastly Object Storage
On-Demand Migration for Fastly Object Storage lets you transfer your data into Fastly Object Storage. This method takes a stateless approach that helps you only migrate your required working set of data to reduce egress charges from your source provider. Here's how it works:
- A client requests an object from your Fastly Object Storage bucket.
- If the object is not yet present, Fastly's Object Storage service reads the asset from a publicly provided source URL (via the
fastly-object-storage-source-urlheader). - The asset is fetched and simultaneously served to the requesting user and asynchronously written into your Fastly Object Storage bucket.
HINT: Because each object is migrated on the first request, implementing Fastly Bot Management can help prevent bots from scraping assets and causing them to migrate before they're actually needed.
Limitations and considerations
On-Demand Migration for Fastly Object Storage does not support objects greater than 250 MB. If an object greater than 250 MB is requested, the response will show that the object is available in the source, but the object won't be copied to the Fastly bucket.
Using On-Demand Migration
To use On-Demand Migration for Fastly Object Storage, you must set a single header, fastly-object-storage-source-url, on a standard GetObject request. The URL must include the following information:
- Source: The initial location of the object, typically a non-Fastly storage provider like AWS S3.
- Destination: The location within Fastly Object Storage where the object will be written.
- (Optional) Pre-signed URLs: The URL must be accessible from Fastly Object Storage. If the object is in a private S3 bucket, use a pre-signed URL to grant temporary, time-limited access to specific objects within an S3 bucket without requiring the recipient to have AWS credentials or permissions.
The following example creates a request with a pre-signed URL for a private, S3-compatible source:
