Here's a short tip: analyze the logs on your origin servers.
If you want to increase the efficiency of your Varnish (or Fastly) cache, you need to figure out what traffic is not cached. By definition, any traffic that reaches your origin is not cached, and thus worthy of investigation. If it reaches your origin, it's either a miss or a pass. You never want misses, but sometimes passes are unwanted, too. You don't want a cookie to cause static files to be passed instead of cached, for instance.
Just generating a top 100 list of URLs requested on your origin goes a long way towards finding quick wins in the form of URLs you assumed were cached, but in reality are not. The common causes, and how to solve them, will be covered in next month's Varnish tip.
One caveat concerning the top 100 approach is randomized cache busters. Used well, cache busters can save a lot of work, but if they're completely random, they'll wreak havoc on your cache. And due to their nature, they might not show up in a top 100 list. So don't look at just a single list, but explore the possibilities of your log analyzer tools. Most will allow you to cut off query strings before the top 100 is made. Some might even allow you to whitelist only specific parameters, which could be useful if you're trying to cache a RESTful API, for instance.
Don't be afraid to immerse yourself in raw logs; sometimes the human brain trumps all other pattern detection.
You may also like:
Introducing Soft Purge, a more efficient way to mark outdated content
Today, we’re excited to announce Soft Purge, a new purging feature that allows you to easily mark content as outdated (stale) instead of permanently deleting it from Fastly’s caches. With Soft Purge, you...
Common causes of a poor cache hit ratio and how to deal with them
The cache hit ratio (or hit ratio for short) is the ratio of hits to cacheable requests (hits and misses combined). There’s also cache coverage, the ratio of cacheable requests to all requests (cacheable requests…
Accelerating Rails, Part 2: Dynamic HTTP Caching
In the second part of our series on accelerating Rails, I’ll cover configuration of a few Fastly features, Varnish and Varnish Configuration Language (VCL), and strategies for caching dynamic content that are targeted towards the…