Varnish and SSL with Pound

Question

I've set up my Varnish server as follows:

backend web1 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}
backend web2 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}
backend web3 {.host = "XXX.XXX.XXX.XXX"; .port = "80";}

backend web1_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}
backend web2_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}
backend web3_ssl {.host = "XXX.XXX.XXX.XXX"; .port = "443";}

director default_director round-robin {
  { .backend = web1; }
  { .backend = web2; }
  { .backend = web3; }
}

director ssl_director round-robin {
  { .backend = web1_ssl; }
  { .backend = web2_ssl; }
  { .backend = web3_ssl; }
}

# Respond to incoming requests.
sub vcl_recv {
# Set the director to cycle between web servers.
set req.grace = 120s;

if (req.http.X-Forwarded-Proto == "https" ) {
   set req.http.X-Forwarded-Port = "443";
   set req.backend = ssl_director;
} else {
   set req.http.X-Forwarded-Port = "80";
   set req.http.X-Forwarded-Proto = "http";
   set req.backend = default_director;
 }

 ...
}

This works perfectly if I hit my IP address (without SSL) in the browser, but if I enable Pound (config below):

ListenHTTPS
    Address XXX.XXX.XXX.XXX #Local IP of the VarnishWebServer
    Port 443
    Cert "/etc/apache2/ssl/apache.pem"
    AddHeader "X-Forwarded-Proto: https"
    HeadRemove "X-Forwarded-Proto"
    Service
            BackEnd
                    Address 127.0.0.1
                    Port 80
            End
    End

End

I get a 503 everyime I try to hit the local IP address (from varnishlog -0):

   11 RxURL        c /favicon.ico
   11 RxProtocol   c HTTP/1.1
   11 RxHeader     c Host: XXX.XXX.XXX (Varnish Server IP Address)
   11 RxHeader     c Connection: keep-alive
   11 RxHeader     c Accept: */*
   11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36
   11 RxHeader     c Accept-Encoding: gzip,deflate,sdch
   11 RxHeader     c Accept-Language: en-US,en;q=0.8
   11 RxHeader     c X-Forwarded-Proto: https
   11 RxHeader     c X-SSL-cipher: DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
   11 RxHeader     c X-Forwarded-For: XXX.XXX.XXX.XXX (My Local machine IP)
   11 VCL_call     c recv lookup
   11 VCL_call     c hash
   11 Hash         c /favicon.ico
   11 Hash         c 198.61.252.81
   11 VCL_return   c hash
   11 VCL_call     c miss fetch
   11 Backend      c 14 ssl_director web2_ssl
   11 FetchError   c http read error: -1 0 (Success)
   11 VCL_call     c error deliver
   11 VCL_call     c deliver deliver
   11 TxProtocol   c HTTP/1.1
   11 TxStatus     c 503
   11 TxResponse   c Service Unavailable
   11 TxHeader     c Server: Varnish
   ...
   11 ReqEnd       c 1175742305 1391779282.930887222 1391779282.934647560 0.000097752 0.003678322 0.000082016
   11 SessionClose c error

I looked at my http listeners and I see this:

 root@machine:/etc/apache2/ssl# lsof -i -n|grep http
 pound     7947 www-data    5u  IPv4  63264      0t0  TCP XXX.XXX.XXX.XXXX:https (LISTEN)
 pound     7948 www-data    5u  IPv4  63264      0t0  TCP XXX.XXX.XXX.XXXX:https (LISTEN)
 varnishd  8333   nobody    7u  IPv4  64977      0t0  TCP *:http (LISTEN)
 varnishd  8333   nobody    8u  IPv6  64978      0t0  TCP *:http (LISTEN)
 varnishd  8333   nobody   13u  IPv4  65029      0t0  TCP XXX.XXX.XXX.XXXX:37493-   >YYYY.YYYY.YYYY.YYYY3:http (CLOSE_WAIT)
 apache2  19433     root    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 apache2  19438 www-data    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 apache2  19439 www-data    3u  IPv4  31020      0t0  TCP *:http-alt (LISTEN)
 pound    19669 www-data    5u  IPv4  31265      0t0  TCP 127.0.0.1:https (LISTEN)
 pound    19670 www-data    5u  IPv4  31265      0t0  TCP 127.0.0.1:https (LISTEN)

Where XXX.XXX.XXX.XXX is the varnish's WebServer's internal IP address, and YYYY.YYYY.YYYY.YYY is the IP address of one of the backend system defined in the VCL.

Any idea why I keep getting 503s?

UPDATE

As noted Varnish doesn't support SSL, so using Pound can transfer the traffic from 443 to 80, but when it's finished - it can't use port 443 (ssl_diretector) to serve the traffic. Removing the ssl_director and making default_director the primary, worked perfectly.

Answer 1

Varnish does not support HTTPS for its backend requests - any communication between Varnish and Apache must be plain HTTP.

What I found works best is to configure Apache to speak plain HTTP on port 443. This allows Apache to generate correct URLs, such as when it needs to redirect the browser.

Here's how you might configure it:

# Listen on port 443, but speak plain HTTP
Listen X.X.X.X:443 http

# Setting HTTPS=on is helpful for ensuring correct behavior of scripting
# languages such as PHP
SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on

<VirtualHost X.X.X.X:443>
    # Specifying "https://" in the ServerName ensures that whenever
    # Apache generates a URL, it uses "https://your.site.com/" instead
    # of "http://your.site.com:443/"
    ServerName https://your.site.com
</VirtualHost>

You will of course need to remove any mod_ssl directives from your Apache configuration.