Direct Fetch for Fastly Object Storage

Direct Fetch for Fastly Object Storage lets you transfer data from a remote source directly into Fastly Object Storage without first downloading it to the client. When a client sends a Direct Fetch request:

1. It sends a PutObject or UploadPart request with the remote object's URL in the fastly-object-storage-source-url header.
2. Fastly Object Storage fetches the data from the provided URL. If the request includes a Range header, the same range is requested from the source.
3. Fastly Object Storage writes the fetched data to the destination object or multipart upload part and returns a standard PutObject or UploadPart response for the requested operation.

Direct Fetch reduces transfer time and client bandwidth when copying objects from another storage provider or assembling a multipart upload from ranges of an existing object.

Before you begin

Make sure you review the prerequisites for using Fastly Object Storage.

Limitations and considerations

Note the following limitations and considerations when using Direct Fetch for Fastly Object Storage.

Direct Fetch requests have the following requirements:

• Direct Fetch supports the PutObject and UploadPart operations.
• The fastly-object-storage-source-url header must be included in the set of signed headers when generating the AWS Sigv4 signature. If the request includes a Range header, it must be included in the signature as well.
• A single Direct Fetch source response body cannot be greater than 5 GB. Use the multipart upload API to transfer a larger object in parts.

The remote source must meet the following conditions:

• The source URL must be reachable by Fastly Object Storage. For an object that isn't publicly accessible, use a pre-signed URL that grants temporary access to the object.
• For a request without a Range header, the source must return HTTP 200 OK. Direct Fetch rejects an unsolicited HTTP 206 Partial Content response.
• For a ranged request, the source must return either HTTP 206 Partial Content with a valid Content-Range header or HTTP 200 OK.
• Multipart ranges are not allowed.
• The source must not require more than one redirect.

Direct Fetch handles transferred data differently than a standard upload:

• The client request must have an empty body and a Content-Length header with a value of 0. Fastly Object Storage obtains the object data from the source URL.
• Custom metadata headers returned from the source in x-amz-meta- prefixed headers will not be written to Fastly Object Storage. To include custom metadata, include the x-amz-meta- prefixed metadata headers on the PutObject or CreateMultipartUpload request. The same metadata restrictions apply to Direct Fetch requests as on standard PutObject or CreateMultipartUpload requests.
• Object metadata such as ETag and Last-Modified for data migrated via Direct Fetch will not match the source ETag or Last-Modified.

Using Direct Fetch with PutObject and UploadPart

To use Direct Fetch for Fastly Object Storage, set the fastly-object-storage-source-url header on a standard PutObject or UploadPart request. Set the header to the URL of the object you want Fastly Object Storage to fetch, typically from a non-Fastly storage provider like AWS S3. If the object is in a private bucket, use a pre-signed URL to grant temporary, time-limited access without sharing the source account's credentials.

Then, create a PutObject request to https://fos-region.object.fastlystorage.app/my-fos-bucket/object-key with an empty body and Content-Length: 0, where:

• fos-region.object.fastlystorage.app is the regional Object Storage endpoint where you want to send the request.
• my-fos-bucket is the name of your Object Storage bucket.
• object-key is the name of the destination object in your Object Storage bucket.

To store only part of the source object, add one supported HTTP byte Range header to the request. The resulting destination object contains only the selected bytes.

Using Direct Fetch with multipart uploads

To transfer an object through multipart upload, first send a standard CreateMultipartUpload request. For each part, send an UploadPart request with the upload ID, part number, fastly-object-storage-source-url header, an empty body, and Content-Length: 0.

You can add a different Range header to each UploadPart request to fetch specific sections of the source object.

After all parts have been uploaded, send a standard CompleteMultipartUpload request with the part numbers and ETags returned by the UploadPart responses.

Validating a transfer

After a successful Direct Fetch PutObject request, send a HeadObject or GetObject request to the same region, bucket, and object key to confirm that the object exists. For multipart uploads, validate the object after the CompleteMultipartUpload request succeeds.

Error handling

Direct Fetch reports feature-specific errors through both an XML response body and a fastly-object-storage-df-error header, which contains a description of the error that occurred. For example, if the source server responds with an unsupported HTTP 503 Service Unavailable status code, the fastly-object-storage-df-error header will report DirectFetchSourceStatus 503 and the response status itself will be HTTP 400 Invalid Request.