Multiple clients into a single HTTP persistent connection to the back-end server

Question

I am working on a web enterprise application serving hundreds of users concurrently. Currently we found that sometimes there are too many TCP connections to our Jetty server and make our server slow.

We heard that some load balancer support TCP offload which consolidates multiple HTTP requests from multiple clients into a single TCP socket to the back-end servers.

We have studied HAProxy, Nginx and Varnish but we cannot find the related features.

I would like to know if there are any load balancing or reverse proxy softwares that support consolidating multiple clients into a single HTTP persistent connection to the back-end server.

Answer 1

There are load balancers out there that do this. The two, that I definitely know can do it, are F5's 'Big-IP LTM' (using the OneConnect feature) and Shaka Technologies 'Ishlangu Load Balancer ADC'.

F5's description for OneConnect:

The BIG-IP system OneConnect feature can increase network throughput by efficiently managing connections created between the BIG-IP system and back-end pool members. The OneConnect feature works with HTTP Keep-Alives to allow the BIG-IP system to minimize the number of server-side TCP connections by making existing connections available for reuse by other clients.

Shaka Technologies description for Connection Multiplexing:

The Ishlangu Load Balancer ADC's connection multiplexing capability improves web application performance and decreases server load by reducing the concurrent connections and connection rate on back-end servers.

Connection Multiplexing reuses TCP connections to each server for multiple clients. With fewer open connections and fewer connections opening/closing results in lower resource consumption per server. Minimizing resource utilization per server enables server consolidation therefore allowing operational cost savings.