Trying to write a proxy server. Content Length MANAGEMENT problem

Question

I am trying to write a proxy server in C language under Linux. It was working fine (I had the perception that it was working fine) until I tried it for streaming media.

Lemme first tell the problem and then I'll jump onto streaming media. To read the incoming data from the website and forward it to the actual client I do this

count = read(websitefd,buffer,BUFSIZ);

write(clientfd,buffer,count);` 

in a continuous while loop until I read up all the data on that socket.

Now the problem is if the actual website sends an HTTP packet with content length field as 1025 bytes and other part of data in other packets then still I always wait for BUFSIZ(8192 bytes) and then I send 8192 bytes to the client machine all together. for normal octet-stream it works fine even though I know its not the right method, because I should forward the packets same as the actual server. So if actual server sends me 2 packet of sizes 1024 and 1024 bytes I send the client a packet of 2048 bytes with the first packet with the HTTP header saying that the content length is 900 bytes (rest all being the http header assuming) but actually I forward a packet of 2048 bytes to client. For Content Type: application/octet-stream it just downloads the whole thing and displays it either as image or html text or asks me to save it.

When the client request for a streaming media, because of the above reason the client is not able to play the video. So what should I do now ? Thanks for reading my question. Please help me out. :)

Answer 1

First, I strongly recommend using an existing proxy server as the base of any proxy system. The HTTP standard is quite complex, much more than you realize. If you are going to implement a proxy server, read RFC 2616 at least three times first.

Second, your proxy server must parse HTTP headers to figure out how much it must send. The three main ways to know how much data to send are as follows:

• If a Content-Length header is present and no Transfer-Encoding header is present: The Content-Length header specifies how much data to relay in bytes. Just go into a loop copying.
• If a Transfer-Encoding: chunked header is present: You must parse the chunked transfer encoding chunk headers. This encoding is frequently used for streaming data, where the total size is not known ahead of time. It's also often used for dynamic data generated by scripts.
• If some other Transfer-Encoding header is present: Close the connection and report a 500 error, unless you know what that encoding is.
• If a Content-Length header is not present, and no Transfer-Encoding header is present: Check for Connection: close (must be present, in HTTP/1.1) and Connection: keep-alive (must NOT be present in HTTP/1.0). If these conditions are violated, trigger a 500 error. Otherwise just keep passing data until the server closes the connection.

I'm deliberately making this a bit vauge - you MUST read the standard if you're implementing a proxy server from scratch, or you will certainly introduce browser incompatibilities and/or security holes! So please, don't do so. Use lighttpd or varnish or something as the core proxy server, and just write a plugin for whatever functionality you need.

Answer 2

I suppose media is transferred in chunks, i.e no Content-Length is present and data is sent until finished. As bdonlan said please read how chunked data works,

And i agree HTTP is pretty nasty (due to many changes and interpretations in time)