Fastly + Scalepost: Extending the Fastly platform to manage AI Crawlers

Tools like ChatGPT, Perplexity, and Claude are reshaping how digital publishers and content creators engage with their audiences. Instead of clicks and page views, entire answers are now generated on the fly, often drawing directly from the content the creators worked hard to produce. As these platforms become go-to channels for news, entertainment, and commerce, the lack of visibility leaves companies uncertain about their true influence and content adoption in a rapidly growing part of the digital publishing landscape.

ScalePost recently introduced a set of tools that gives publishers and brands clear visibility into how their content appears in AI answers: what’s cited, by whom, and how often, so they can measure presence and improve it. These tools are currently utilized by revenue, marketing, and SEO teams. 

But before diving deeper into how the solution works, it’s worth stepping back to understand why the Fastly platform is uniquely suited for innovation like this. Built to be extended, customized, and enhanced, Fastly gives developers and partners the foundation to create new capabilities at the edge, making solutions like ScalePost possible in the first place.

How Fastly Delivers Unmatched Visibility and Extensibility

Developers and third-party companies continuously enhance the value of Fastly, thanks to our flexible, developer-friendly, and highly customizable architecture. Online businesses can run services and applications directly at the edge, delivering immediate performance improvements and gain greater control over how traffic is managed. This makes Fastly not just another CDN, but an edge platform that grows alongside its users and strengthens their investments over time.

A key differentiator is the visibility Fastly provides. With instant and complimentary log files, companies gain a true real-time data source that shows exactly what’s happening across their traffic. ScalePost, for example, leverages this capability to provide its customers with an accurate and instant view of how content is being accessed and referenced by AI platforms. Where other providers only offer a fraction of log data, often with significant delays, and often at a cost, Fastly makes 100% of customer logs available immediately. That level of transparency empowers businesses to detect issues faster, unlock deeper insights, and build solutions their customers can trust. It’s this combination of openness, speed, and completeness that makes Fastly the strongest complement to any CDN strategy.

ScalePost: Restoring Visibility and Regaining Control

In recent weeks, we’ve looked at a range of solutions built on the Fastly platform aimed at helping content owners capture revenue when their work is scraped. New to this category is ScalePost, which measures how content or brands are cited and positioned in AI answers, then guides publishers or brand owners on how to expand their presence where it matters.

Traditional analytics only measure human referrals and pageviews; they don’t reveal whether an article or brand is powering an AI-generated answer. Once you understand how LLM platforms are using your content and brand, you can make informed decisions: optimize Answer Engine Optimization (AEO) visibility by shaping the content and formats most likely to surface in AI answers; explore monetization opportunities where licensing and partnerships are active today, and where demand is likely to grow; and control access using rules to block unwanted crawlers or set the terms under which they reach your content.

ScalePost delivers actionable signals directly into existing workflows. Integration APIs expose a company’s “AI search footprint,” showing which URLs, headlines, or entities are being lifted into LLM answers. Classification data enables fine-grained policy enforcement, determining which bots are allowed, throttled, or blocked at the edge, while performance metrics and AEO insights inform optimization strategies. By combining Fastly’s edge-native visibility - built on architecture designed from the ground up to run at the network edge to deliver real-time insight - with ScalePost’s correlation layer, Fastly customers gain an instant, reliable, developer-friendly way to detect, classify, and manage AI bot traffic as part of their broader infrastructure strategy.

To sum up, Scalepost provides the following features:

• Visibility into your AI search footprint: Spot when, where, and how your articles, headlines, and brands are appearing across popular chatbots. This is a core AEO signal.

• Mentions and context: See which content or brands perform best in AI answers, and how you compare against competitors or peers.

• Actionable insights for AEO: Go beyond pageviews and clicks to understand how AI platforms are shaping reach and influencing brand perception.

Shedding Light on Hidden Infrastructure Costs

ScalePost also helps engineering and operations teams explain rising infrastructure costs that traditional analytics overlook. AI bots crawling at scale can significantly increase bandwidth, origin egress, and compute utilization, but because this traffic isn’t tied to human sessions, it never appears in referral or revenue reports. The result is a widening gap between infrastructure spend and measurable return. By classifying and correlating AI bot requests with edge-level logs, ScalePost exposes the true drivers of increasing infrastructure cost. Teams can then quantify non-revenue traffic, tune caching and delivery policies, and decide when to throttle, block, or license AI consumption. Put another way: ScalePost turns unexplained costs into measurable activity that can be actively managed.

Fastly’s HTTPS real-time log streaming makes it simple to activate ScalePost, with no impact on human visits. Once enabled, you’ll have immediate access to a real-time dashboard and historical trends, giving you clear visibility into how AI interactions with your content evolve over time.

For more information, visit scalepost.ai and Fastly’s AI bot management pages.