Direct Fetch for Fastly Object Storage
Direct Fetch for Fastly Object Storage lets you transfer data from a remote source directly into Fastly Object Storage without first downloading it to the client. When a client sends a Direct Fetch request:
- It sends a
PutObjectorUploadPartrequest with the remote object's URL in thefastly-object-storage-source-urlheader. - Fastly Object Storage fetches the data from the provided URL. If the request includes a
Rangeheader, the same range is requested from the source. - Fastly Object Storage writes the fetched data to the destination object or multipart upload part and returns a standard
PutObjectorUploadPartresponse for the requested operation.
Direct Fetch reduces transfer time and client bandwidth when copying objects from another storage provider or assembling a multipart upload from ranges of an existing object.
Before you begin
Make sure you review the prerequisites for using Fastly Object Storage.
Limitations and considerations
Note the following limitations and considerations when using Direct Fetch for Fastly Object Storage.
Direct Fetch requests have the following requirements:
- Direct Fetch supports the
PutObjectandUploadPartoperations. - The
fastly-object-storage-source-urlheader must be included in the set of signed headers when generating the AWS Sigv4 signature. If the request includes aRangeheader, it must be included in the signature as well. - A single Direct Fetch source response body cannot be greater than 5 GB. Use the multipart upload API to transfer a larger object in parts.
- A maximum of 16 additional headers may be passed to the source.
The remote source must meet the following conditions:
- The source URL must be reachable by Fastly Object Storage. For an object that isn't publicly accessible, use a pre-signed URL that grants temporary access to the object or pass authentication headers to the source.
- For a request without a
Rangeheader, the source must returnHTTP 200 OK. Direct Fetch rejects an unsolicitedHTTP 206 Partial Contentresponse. - For a ranged request, the source must return either
HTTP 206 Partial Contentwith a validContent-Rangeheader orHTTP 200 OK. - Multipart ranges are not allowed.
- The source must not require more than one redirect.
Direct Fetch handles transferred data differently than a standard upload:
- The client request must have an empty body and a
Content-Lengthheader with a value of0. Fastly Object Storage obtains the object data from the source URL. - The same metadata restrictions apply to Direct Fetch requests as to standard
PutObjectorCreateMultipartUploadrequests. For copying and overriding metadata, see Working with object metadata. - Object metadata such as
ETagandLast-Modifiedfor data migrated via Direct Fetch will not match the sourceETagorLast-Modified.
Using Direct Fetch with PutObject and UploadPart
To use Direct Fetch for Fastly Object Storage, set the fastly-object-storage-source-url header on a standard PutObject or UploadPart request. Set the header to the URL of the object you want Fastly Object Storage to fetch, typically from a non-Fastly storage provider like AWS S3. If the object is in a private bucket, use a pre-signed URL to grant temporary, time-limited access without sharing the source account's credentials.
IMPORTANT: The fastly-object-storage-source-url header must be included in the set of signed headers when generating the AWS Sigv4 signature. The Range header must also be signed when requesting a range from the source.
Then, create a PutObject request to https://fos-region.object.fastlystorage.app/my-fos-bucket/object-key with an empty body and Content-Length: 0, where:
fos-region.object.fastlystorage.appis the regional Object Storage endpoint where you want to send the request.my-fos-bucketis the name of your Object Storage bucket.object-keyis the name of the destination object in your Object Storage bucket.
To store only part of the source object, add one supported HTTP byte Range header to the request. The resulting destination object contains only the selected bytes.
Using Direct Fetch with multipart uploads
To transfer an object through multipart upload, first send a standard CreateMultipartUpload request. Set any custom metadata and HTTP metadata you want on the destination object on this request. Direct Fetch doesn't copy metadata from the source for multipart uploads.
For each part, send an UploadPart request with the upload ID, part number, fastly-object-storage-source-url header, an empty body, and Content-Length: 0. You can also pass additional headers to the source on each UploadPart request.
IMPORTANT: The fastly-object-storage-source-url header must be included in the set of signed headers when generating the AWS Sigv4 signature. The Range header must also be signed when requesting a range from the source.
You can add a different Range header to each UploadPart request to fetch specific sections of the source object.
After all parts have been uploaded, send a standard CompleteMultipartUpload request with the part numbers and ETags returned by the UploadPart responses.
Passing headers to the source
To send additional headers to the source URL, include the fastly-object-storage-source-header header on your PutObject or UploadPart request. Use the format fastly-object-storage-source-header: header-name=header-value. Repeat this header for each header you want to send to the source, up to 16 per request.
For example, if the source requires a bearer token for authentication and a custom X-Tenant-ID header to identify the tenant, include the following headers. Replace YOUR_SOURCE_TOKEN and example-tenant with values for your source:
fastly-object-storage-source-url: https://source.example.com/object-keyfastly-object-storage-source-header: authorization=Bearer YOUR_SOURCE_TOKENfastly-object-storage-source-header: x-tenant-id=example-tenantFastly Object Storage sends Authorization: Bearer YOUR_SOURCE_TOKEN and X-Tenant-ID: example-tenant to the source URL.
NOTE: Send each source header as a separate fastly-object-storage-source-header: header-name=header-value field. Do not combine entries into a comma-separated list. Fastly Object Storage treats commas as part of the source header's value, not as separators between entries. Your HTTP client and any intermediaries must preserve these as separate fields.
Working with object metadata
For PutObject requests, Direct Fetch copies the following HTTP metadata headers from the source: Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Type, and Expires. If you set any of these headers on the incoming PutObject request, that value replaces the corresponding source value on the destination object.
To control how Direct Fetch handles custom metadata on a PutObject request, set the x-amz-metadata-directive header:
| Value | Custom metadata written to the destination object |
|---|---|
Not set or COPY | Copies x-amz-meta-* headers from the source. |
REPLACE | Uses only the x-amz-meta-* headers set on the incoming PutObject request. |
The x-amz-metadata-directive header controls only custom metadata, not the HTTP metadata headers listed above.
For example, to replace custom metadata and override the source's Cache-Control, include these headers on your Direct Fetch PutObject request:
x-amz-metadata-directive: REPLACEx-amz-meta-project: example-projectCache-Control: max-age=3600The destination object uses only x-amz-meta-project as custom metadata and has Cache-Control: max-age=3600. Direct Fetch copies the other HTTP metadata headers listed above from the source.
Validating a transfer
After a successful Direct Fetch PutObject request, send a HeadObject or GetObject request to the same region, bucket, and object key to confirm that the object exists. For multipart uploads, validate the object after the CompleteMultipartUpload request succeeds.
Error handling
Direct Fetch reports feature-specific errors through both an XML response body and a fastly-object-storage-df-error header, which contains a description of the error that occurred. For example, if the source server responds with an unsupported HTTP 503 Service Unavailable status code, the fastly-object-storage-df-error header will report DirectFetchSourceStatus 503 and the response status itself will be HTTP 400 Invalid Request.