Hot reconfiguration of HAProxy still lead to failed request, any suggestions?

Question

I found there are still failed request when the traffic is high using command like this

haproxy -f /etc/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

to hot reload the updated config file.

Here below is the presure testing result using webbench :

/usr/local/bin/webbench -c 10 -t 30 targetHProxyIP:1080
Webbench – Simple Web Benchmark 1.5
Copyright (c) Radim Kolar 1997-2004, GPL Open Source Software.

Benchmarking: GET targetHProxyIP:1080
10 clients, running 30 sec.

Speed=70586 pages/min, 13372974 bytes/sec.
**Requests: 35289 susceed, 4 failed.**

I run command

haproxy -f /etc/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

several times during the pressure testing.

In the haproxy documentation, it mentioned

They will receive the SIGTTOU 611 signal to ask them to temporarily stop listening to the ports so that the new 612 process can grab them

so there is a time period that the old process is not listening on the PORT(say 80) and the new process haven’t start to listen to the PORT (say 80), and during this specific time period, it will cause the NEW connections failed, make sense?

So is there any approach that makes the configuration reload of haproxy that will not impact both existing connections and new connections?

Answer 1

On recent kernels where SO_REUSEPORT is finally implemented (3.9+), this dead period does not exist anymore. While a patch has been available for older kernels for something like 10 years, it's obvious that many users cannot patch their kernels. If your system is more recent, then the new process will succeed its attempt to bind() before asking the previous one to release the port, then there's a period where both processes are bound to the port instead of no process.

There is still a very tiny possibility that a connection arrived in the leaving process' queue at the moment it closes it. There is no reliable way to stop this from happening though.