How to get cache tags invalidation working with Varnish Purger

Question

On my server, I've got Varnish 5.2 working, so far so good. However, any changes made on the site are not shown to anonymous visitors, until the cache expires.

I installed the purge module together with the Varnish Purge module to get cache tags invalidation going. Then added a purger there, and another, and another, but no matter which settings I try, the requests keep on being served from cache.

Manually purging works (I am testing on node 1042), like this:

curl -X BAN http://127.0.0.1:6081/ -H "Cache-Tags: node:1042"

However, when I activate purging through cron, there's this error:

purger_varnish_84f7afee13: item failed due GuzzleHttp\Exception\ConnectException, details (JSON): {"msg":"cURL error 28: Operation timed out after 1000 milliseconds with 0 out of 0 bytes received (see https:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)","uri":"https:\/\/127.0.0.1:6081\/","method":"BAN","guzzle_opt":{"http_errors":true,"connect_timeout":1,"timeout":1,"verify":false},"headers":{"user-agent":"varnish_purger module for Drupal 8.","cache-tags":"config:contact.form.personal"}}

Obviously, I'm doing something wrong, but what?

In confusion, I tried all of these purger settings without joy:

Hostname: 127.0.0.1  Port: 443  Path: /  Request Method: BAN  Scheme https, Verify SSL certificate

127.0.0.1 Port: 80 http

127.0.0.1 Port: 8080 http

localhost Port: 6081 http

localhost Port: 80 http

localhost Port: 6081 https, Verify SSL certificate

127.0.0.1 Port: 8080 https

localhost Port: 8080 https

The contents of /etc/varnish/usr.vcl:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

acl purge {
"127.0.0.1";
}

# Respond to incoming requests.
sub vcl_recv {
# Add an X-Forwarded-For header with the client IP address.
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
}
else {
set req.http.X-Forwarded-For = client.ip;
}
}
# Only allow PURGE requests from IP addresses in the 'purge' ACL.
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "Not allowed."));
}
return (hash);
}
# Only allow BAN requests from IP addresses in the 'purge' ACL.
if (req.method == "BAN") {
# Same ACL check as above:
if (!client.ip ~ purge) {
return (synth(403, "Not allowed."));
}
# Logic for the ban, using the Cache-Tags header. For more info
# see https://github.com/geerlingguy/drupal-vm/issues/397.
if (req.http.Cache-Tags) {
ban("obj.http.Cache-Tags ~ " + req.http.Cache-Tags);
}
else {
return (synth(403, "Cache-Tags header missing."));
}
# Throw a synthetic page so the request won't go to the backend.
return (synth(200, "Ban added."));
}
if (req.method == "URIBAN") {
    ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
    # Throw a synthetic page so the request won't go to the backend.
    return (synth(200, "Ban added."));
  }
# Only cache GET and HEAD requests (pass through POST requests).
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Pass through any administrative or AJAX-related paths.
if (req.url ~ "^/status.php$" ||
req.url ~ "^/update.php$" ||
req.url ~ "^/admin$" ||
req.url ~ "^/admin/.*$" ||
req.url ~ "^/flag/.*$" ||
req.url ~ "^.*/ajax/.*$" ||
req.url ~ "^.*/ahah/.*$") {
return (pass);
}

# Removing cookies for static content so Varnish caches these files.
if (req.url ~ "(?i).(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)\?$") {
unset req.http.Cookie;
}


if (req.http.Cookie) {

    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
      unset req.http.Cookie;
}
    else {
      return (pass);
    }
}


}
# Set a header to track a cache HITs and MISSes.
sub vcl_deliver {
# Remove ban-lurker friendly custom headers when delivering to client.
unset resp.http.X-Url;
unset resp.http.X-Host;
# Comment these for easier Drupal cache tag debugging in development.
#unset resp.http.Cache-Tags;
#unset resp.http.X-Drupal-Cache-Contexts;
if (obj.hits > 0) {
set resp.http.Cache-Tags = "HIT";
}
else {
set resp.http.Cache-Tags = "MISS";
}
}
# Instruct Varnish what to do in the case of certain backend responses (beresp).
sub vcl_backend_response {
# Set ban-lurker friendly custom headers.
set beresp.http.X-Url = bereq.url;
set beresp.http.X-Host = bereq.http.host;
# Cache 404s, 301s, at 500s with a short lifetime to protect the backend.
if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500) {
set beresp.ttl = 10m;
}
# Don't allow static files to set cookies.
# (?i) denotes case insensitive in PCRE (perl compatible regular expressions).
# This list of extensions appears twice, once here and again in vcl_recv so
# make sure you edit both and keep them equal.

if (bereq.url ~ "(?i).(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") {
unset beresp.http.set-cookie;
}
# Allow items to remain in cache up to 6 hours past their cache expiration.
set beresp.grace = 6h;
}

The contents of the nginx server configuration:

server {
    listen 443 ssl http2;
    server_name test.example.com;
    port_in_redirect off;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
      proxy_pass http://127.0.0.1:6081;
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header HTTPS "on";
      proxy_set_header If-Modified-Since $http_if_modified_since;
      proxy_buffering on;
      proxy_buffer_size   128k;
      proxy_buffers   4 256k;
      proxy_busy_buffers_size   256k;
    }
}

server {
   listen 8080;
   server_name test.example.com;
   root /home/example/domains/test/public_html/web;
   index index.php index.html index.htm index.nginx-debian.html;
   port_in_redirect off;

   location / {
      try_files $uri $uri/ /index.php?$query_string;
   }

   location ~ \.php$ {
      include snippets/fastcgi-php.conf;
      fastcgi_pass 127.0.0.1:9000;
   }
}

server {
    listen 80;
    if ($host = test.example.com) {
        return 301 https://$host$request_uri;
    }
    server_name test.example.com;
    return 404;
}

Answer 1

So, it was an error in the settings of the purger from the purge module. That became clear after asking this question on Stackoverflow.

Automated Cron or Drush were not able to connect to Varnish on port 6081 over TLS. The purger had to be changed to use plain HTTP In the Drupal UI I changed the purger with host 127.0.0.1 and port 6081 on http at /admin/config/development/performance/purge That solved it!