You appear to be offline. Some site functionality may not work.

Analyze your origin logs to get a higher cache efficiency

By  Rogier Mulhuijzen, Senior Professional Services Engineer, January 27, 2015 in VarnishPerformanceEdge

Here's a short tip: analyze the logs on your origin servers.

If you want to increase the efficiency of your Varnish (or Fastly) cache, you need to figure out what traffic is not cached. By definition, any traffic that reaches your origin is not cached, and thus worthy of investigation. If it reaches your origin, it's either a miss or a pass. You never want misses, but sometimes passes are unwanted, too. You don't want a cookie to cause static files to be passed instead of cached, for instance.

Just generating a top 100 list of URLs requested on your origin goes a long way towards finding quick wins in the form of URLs you assumed were cached, but in reality are not. The common causes, and how to solve them, will be covered in next month's Varnish tip.

Commonly used offline log analysis tools are AWStats and Piwik. There are also plenty of online (live) log analysis tools like Logentries and Splunk.

One caveat concerning the top 100 approach is randomized cache busters. Used well, cache busters can save a lot of work, but if they're completely random, they'll wreak havoc on your cache. And due to their nature, they might not show up in a top 100 list. So don't look at just a single list, but explore the possibilities of your log analyzer tools. Most will allow you to cut off query strings before the top 100 is made. Some might even allow you to whitelist only specific parameters, which could be useful if you're trying to cache a RESTful API, for instance.

Don't be afraid to immerse yourself in raw logs; sometimes the human brain trumps all other pattern detection.

Varnish Performance Edge

You may also like:

Subscribe to our newsletter

Introducing Soft Purge, a more efficient way to mark outdated content

Today, we’re excited to announce Soft Purge, a new purging feature that allows you to easily mark content as outdated (stale) instead of permanently deleting it from Fastly’s caches. With Soft Purge, you...

Common causes of a poor cache hit ratio and how to deal with them

The cache hit ratio (or hit ratio for short) is the ratio of hits to cacheable requests (hits and misses combined). There’s also cache coverage, the ratio of cacheable requests to all requests (cacheable requests…

Accelerating Rails, Part 2: Dynamic HTTP Caching

In the second part of our series on accelerating Rails, I’ll cover configuration of a few Fastly features, Varnish and Varnish Configuration Language (VCL), and strategies for caching dynamic content that are targeted towards the…


Rogier Mulhuijzen | Senior Professional Services Engineer

Rogier “Doc” Mulhuijzen is a senior professional services engineer and Varnish wizard at Fastly, where performance tuning and troubleshooting have formed the foundation of his 18-year career. When he’s not helping customers, he sits on the Varnish Governance Board, where he helps give direction and solve issues for the Varnish open source project. In his spare time, he likes to conquer all terrains by riding motorcycles, snowboarding, and sailing.