Varnish ESI recognized but not included

Question

Hey everybody. I'm using Edge Side Includes on Varnish to cache a static HTML shell, then include some dynamic HTML and a JSON object.

The ESI for HTML is getting included properly, but the ESI for the JSON object is not being included; Debug statements establish that Varnish sees it, but it inexplicably decides not to include the JSON ESI.

Here's the loading structure:

page uncached. blank page with headers to set a cookie and esi 1

    esi 1 cached.  A bunch of static HTML, plus esi 2 and esi 3

        esi 2 uncached.  A JSON object which is different on every page load.

        esi 3 uncached.  Some HTML which updates with every page load.  

Again, esi 1 and esi 3 work just fine, but esi 2 is not getting included. The content at esi 2 is just this:

{session_page_history_id:24231}

I read that ESI code which doesn't look like XML causes errors, so I set the runtime parameter esi_syntax=0x7 . No dice, but at least I'm getting some debug data now.

The broken call for esi 2 looks like this:

<esi:include src="http://www.domain.com/dynamic_esi/print_variables?esi__desired_var_names[]=session_page_history_id&for_esi=1"/>

And the (working) call for esi 3 looks like this:

<esi:include src="http://www.domain.com/dynamic_esi/get_service_magic_html?&for_esi=1"/>

As I mentioned, Varnish appears to recognize that I'm trying to make an ESI include, but this is the only relevant Debug message I get from varnishlog:

   25 Debug        c AddBit: 6997 <<![CDATA[

window.esi__values =

<esi:include src="http://www.domain.com/dynamic_esi/print_variables?esi__desired_var_names[]=session_page_history_id&for_esi=1"/>
;

 //]]>
 </script>
 <link rel="stylesheet" href="http://www.

The output seen by a client contains all the normal content, but with the <esi> tag left intact. It looks just like what you see in the above Debug message.

Does anyone know why esi 2 isn't getting parsed/included? Any help would be GREATLY appreciated!

Answer 1

So apparently the ESI tag wasn't getting processed because it was inside a //[CDATA[ block. I'm sure this has something to do with XML compliance. I thought that setting esi_syntax=7 would prevent such a problem, but it turns out that that only affects the content of the ESI itself (it doesn't lift the safety checks for the code which does the including).

The solution will be to rework my application so that the ESI is its own self-contained script tag.

Answer 2

Try adding a backslash before the "&" in the querystring.

At least with me Varnish ignores all parameters but the first one if you don't do it.

E.g.:

Maybe you'll have problems with the "[" and "]" and they will also need a backslash.