Can I accomplish this in Varnish?

Question

I have a website that fetches news articles via RSS feeds from many different sites. The script adds those articles to a mysql db. I display 10 articles per page over many many pages to the viewers.

Currently I am caching full pages and purging the cache every time the update script adds new articles to the database. It's fast but not very efficient. Pages cached only have a lifespan of a few minutes, even though the articles themselves can be cached for a longer time.

Can I use varnish to store the individual articles on cache? So every page generated would have links/pointers/references to the articles, instead of having the articles themselves in there. I can then cache those pages for a short period of time while caching the articles for much longer.

How would I go about doing this on Varnish or any other caching mechanism? Would that perform worse/same as full page caches on cache hit?

I'm on Ubuntu 12.04 and nginx.

Thanks!

Answer 1

You want to use edge side includes. Basically, your pages use ESI links like this:

<div class="articles">
  <esi:include src="/path/to/article/1"/>
  <esi:include src="/path/to/article/2"/>
  <esi:include src="/path/to/article/3"/>
</div>

Now in Varnish, have this:

sub vcl_fetch {
  set beresp.do_esi = true;
}

or in Varnish 4:

sub vcl_backend_response {
  set beresp.do_esi = true;
}

When the above code goes through Varnish, Varnish will replace each esi:include with the contents of the URL /path/to/article/1. You can cache the page itself for a few minutes, and each article for days.