Are there any restrictions in writing multiple http responses?

Question

I am building a HTTP proxy with netty, which supports HTTP pipelining. Therefore I receive multiple HttpRequest Objects on a single Channel and got the matching HttpResponse Objects. The order of the HttpResponse writes is the same than I got the HttpRequest. If a HttpResponse was written, the next one will be written when the HttpProxyHandler receives a writeComplete event.

The Pipeline should be convenient:

final ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("writer", new HttpResponseWriteDelayHandler());
pipeline.addLast("deflater", new HttpContentCompressor(9));
pipeline.addLast("handler", new HttpProxyHandler());

Regarding this question only the order of the write calls should be important, but to be sure I build another Handler (HttpResponseWriteDelayHandler) which suppresses the writeComplete event until the whole response was written.

To test this I enabled network.http.proxy.pipelining in Firefox and visited a page with many images and connections (a news page). The problem is, that the browser does not receive some responses in spite of the logs of the proxy consider them as sent successfully.

I have some findings:

• The problem only occurs if the connection from proxy to server is faster than the connection from proxy to browser.
• The problem occurs more often after sending a larger image on that connection, e.g. 20kB
• The problem does not occur if only 304 - Not Modified responses were sent (refreshing the page considering browser cache)
• Setting bootstrap.setOption("sendBufferSize", 1048576); or above does not help
• Sleeping a timeframe dependent on the responses body size in before sending the writeComplete event in HttpResponseWriteDelayHandler solves the problem, but is a very bad solution.

Answer 1

I found the solution and want to share it, if anyone else has a similar problem:

The content of the HttpResponse is too big. To analyze the content the whole HTML document was in the buffer. This must be splitted in Chunks again to send it properly. If the HttpResponse is not chunked I wrote a simple solution to do it. One needs to put a ChunkedWriteHandler next to the logic handler and write this class instead of the response itself:

public class ChunkedHttpResponse implements ChunkedInput {

    private final static int       CHUNK_SIZE = 8196;
    private final HttpResponse     response;
    private final Queue<HttpChunk> chunks;
    private boolean                isResponseWritten;

    public ChunkedHttpResponse(final HttpResponse response) {
        if (response.isChunked())
            throw new IllegalArgumentException("response must not be chunked");

        this.chunks = new LinkedList<HttpChunk>();
        this.response = response;
        this.isResponseWritten = false;

        if (response.getContent().readableBytes() > CHUNK_SIZE) {
            while (CHUNK_SIZE < response.getContent().readableBytes()) {
                chunks.add(new DefaultHttpChunk(response.getContent().readSlice(CHUNK_SIZE)));
            }
            chunks.add(new DefaultHttpChunk(response.getContent().readSlice(response.getContent().readableBytes())));
            chunks.add(HttpChunk.LAST_CHUNK);

            response.setContent(ChannelBuffers.EMPTY_BUFFER);
            response.setChunked(true);
            response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING, HttpHeaders.Values.CHUNKED);
        }
    }

    @Override
    public boolean hasNextChunk() throws Exception {
        return !isResponseWritten || !chunks.isEmpty();
    }

    @Override
    public Object nextChunk() throws Exception {
        if (!isResponseWritten) {
            isResponseWritten = true;
            return response;
        } else {
            HttpChunk chunk = chunks.poll();
            return chunk;
        }
    }

    @Override
    public boolean isEndOfInput() throws Exception {
        return isResponseWritten && chunks.isEmpty();
    }

    @Override
    public void close() {}
}

Then one can call just channel.write(new ChunkedHttpResponse(response) and the chunking is done automatically if needed.