Performance improvement in Tunneling HTTP over TCP

Question

I have a design requirement which requires HTTP communication over underlying TCP networking module

[HTTP_CLIENT<--->LOCAL_TCP_PSEUDOSERVER]---------TCP/IP--------[LOCAL_TCP_PSEUDOSERVER<--->HTTP_SERVER]

1. HTTP cllient talks to local running TCP pseudoserver
2. This pseudoserver communicates to TCP pseudoserver which in turn talks to HTTP_SERVER

The overall functionality works fine as expected, but performance is too low.

After some research, i would few solutions, below are the same

a) HTTP is request response packets have small data packets. This data has to travel over TCP, which enables the Nagle algorithm by default So we applied TCP_NODELAY to disable the Nagle algorithm so that the small data packets are delivered to the remote host without delay.

b) Using HTTP keep-alive. This will make the TCP socket open so that another request can be made without setting up a new connection.

Updated query: Inshort, how does http proxy server work so efficeintly?

FYI: Here is the info I referred

Optimizing HTTP: Keep-alive and Pipelining

Design issues - Sending small data segments over TCP with Winsock

Network Performance Effects of HTTP/1.1, CSS1, and PNG

HTTP Pipelining – Big in Mobile

The above experiment improved the performance at one level, but I suppose there is huge scope of improvement. Is the approach correct? Also, what other parameters may impact performance?

Update 2

consider A-B-E-F. A-B , E-F are local loopback connection, where A is http client, B is pseudo server. E is pseudo server, F is HTTP server. B-E is simple TCP socket communication.

A-B, E-F communicates is loopback and reuses same port

int on = 1; setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)); bind(s, (struct sockaddr *) &sin, sizeof (sin));

Performance is overall slow between A-F

Please share your valuable inputs

Many Thanks

Answer 1

You can have N layers of proxying without enduring N times the latency. Clearly you have got the design and implementation of your 'pseudo-servers' severely wrong. There's no reason to write your own, and most people who try it seem to make a complete hash of it judging by what I see here and in other comparable forums.

These 'pseudo-servers' are really just HTTP proxies, and there's no reason why you shouldn't use a proper one, such as Apache Squid or even Apache HTTPD.

If you post some code you may get some detailed comments.