arXiv Scales and Secures the Foundation of Open Science at the Edge with Fastly
Key results
Managed record traffic spikes of up to 650 million CDN requests seamlessly.
Blocked more than 150 million malicious attack requests in a single month with Next-Gen WAF.
Replaced manual log analysis with real-time rate limiting to defend against sophisticated scripts and AI crawlers.
Accelerated the migration from legacy university infrastructure to the cloud with advanced edge routing.
Reduced operational costs through seamless, discounted Google Cloud integrations as part of the Fastly Fast Forward program.
The challenge
arXiv is a foundational pillar and pioneer of the open access science community, providing researchers with access to nearly 3.1 million papers across physics, computer science, mathematics, and beyond. With 31,604 new submissions in May 2026 alone, the platform is experiencing rapid, sustained growth in traffic. This surge is heavily driven by the explosion of AI and large language models (LLMs), with an increasing number of automated scripts, crawlers, and bots hitting the site to extract data.
Prior to signing with Fastly, the lean engineering team at arXiv battled overwhelmed servers and late-night operational alerts. Determining what was legitimate traffic versus malicious or aggressive automated requests proved difficult.The team relied on custom, internal software to identify and block problematic IP addresses, falling back on manual intervention when that software couldn't keep pace with modern, distributed traffic bursts. arXiv needed a robust solution to manage load without accidentally blocking vital indexers like Google.
The solution
As a member of Fastly's Fast Forward program—which empowers open source and mission-driven organizations—arXiv implemented Fastly's CDN and Next-Gen WAF to automate infrastructure and regain control over their global traffic.
Seamless cloud migration and routing: As arXiv transitions into an independent nonprofit organization and migrates its infrastructure from Cornell University to the cloud, Fastly acts as the critical traffic director. The team routes traffic precisely based on endpoints and cookies, cleanly decoupling backend systems to ensure a friction-free migration.
Cache read-only data at the edge: arXiv shifted its vast repository of read-only research data behind Fastly CDN servers. This architectural shift instantly made exponential traffic manageable and effortlessly absorbs record-breaking daily surges.
Proactively neutralize malicious traffic at scale: Fastly's Next-Gen WAF has been instrumental in blocking over 150 million malicious attack requests in a single month. By identifying and stopping sophisticated threats, such as high-frequency bot attacks and volumetric abuse, arXiv's systems stay protected from harmful requests, ensuring uninterrupted service for legitimate academic users.
Reduce unwanted volumetric load: With the integration of Next-Gen WAF, arXiv replaced manual log reviews with automated, real-time rate limiting. The Next-Gen WAF easily detects and blocks bug bounty hunters, vulnerability scanners, and aggressive AI scripts before they burden the origin servers.
Protect user privacy: To maintain a collaborative environment, arXiv allows users to view author email addresses for legitimate academic purposes. Working with Fastly, arXiv implemented edge logic to restrict aggressive email scraping, preventing bad actors from building spam databases while still enabling meaningful peer-to-peer academic contact.
Maximize Google Cloud integrations: Fastly's deep integration with GCP allows arXiv to stream logs directly to Google Cloud Storage. This setup is highly cost-effective and simple to configure, saving valuable developer time.
Key takeaway
For a vital open science platform managing billions of requests and 5 million active monthly users, infrastructure automation is essential. Fastly acts as an extension of the arXiv engineering team, handling millions of complex security decisions automatically.
“All of these little, individual, specific cases, the WAF has been really great at finding,” noted Brian Maltzan, Senior Software Engineer at arXiv. “These aren't things that we would have enough staff resources, enough staff time to investigate and figure out”.
As arXiv continues to scale its infrastructure into an independent cloud environment, Fastly remains integral to its operational stability. The team plans to explore advanced API key management to further refine access controls, ensuring that the world's most critical research remains open, secure, and highly available for the next generation of scientific discovery.