Today, we are thrilled to announce Direct Fetch, a powerful new S3 server side transfer API designed to help you actively transfer data directly from an S3-compatible source provider into Fastly Object Storage.
The Bottleneck of Traditional S3 Copies
Normally, migrating data with an S3 compatible client like rclone means downloading objects to your local machine and then uploading them to their new destination. This conventional method suffers from a few limitations:
Double Data Movement: You are forced to download the data and then upload it through a client application.
Client Limitations: Your transfer speed is limited by your local client's compute power and bandwidth constraints.
Public Internet Hops: The data often travels over the unpredictable public internet, increasing latency and cost.
How Direct Fetch Changes the Game
Direct Fetch takes an optimized approach that completely bypasses the need to download and upload data through a client. Compared to Fastly’s On-Demand Migration which triggers passively when a user requests an object, Direct Fetch is an active write operation. Any client can make a standard PutObject or UploadPart request with an empty payload (Content-Length: 0), including the URL of the source data in the fastly-object-storage-source-url header instead. Fastly fetches the object directly from the URL and synchronously writes it into your bucket.
Key advantages include:
Massive Speed: Transfers happen directly over Fastly’s highly optimized network, skipping the open internet and client bottlenecks.
Virtually Limitless Scale: A single client cannot scale past its own hardware limitations. Direct Fetch; however, leverages the immense capacity of Fastly's edge cloud, allowing your transfers to scale seamlessly up to the massive limits of Fastly's global bandwidth and compute capabilities.
Minimal Client Compute: Your client simply orchestrates the move via command without needing to process the actual data payload.
Secure Migrations: You can use pre-signed URLs in the custom header to migrate private assets without ever exposing long-term credentials.
Direct Fetch also supports partial transfers: for example, when using UploadPart for multipart uploads, you can include a standard Range header (e.g., bytes=0-41943039) to fetch and store just the specified byte range as that part. Note that the source URL must return an HTTP 200 OK or 206 Partial Content for the copy to proceed.
Start Migrating Today
Direct Fetch is available now for anyone moving data into Fastly Object Storage. Set the fastly-object-storage-source-url header on a PutObject or UploadPart request and Fastly handles the transfer server-side.
Check out our documentation for more information. If you’re planning a large-scale migration and need additional assistance, don’t hesitate to reach out to your Fastly account team to talk through the details.