Reading stream over TCP on a SocketChannel with undefined number of Bytes

Question

I am trying to read a stream on a SocketChannel without defining the number of bytes.

The alternate solution i thought about is storing different ByteBuffers of a pre-defined size into a list which will allow me afterwards to allocate a new ByteBuffer of the received size and put the result inside.

The problem is that i am on blocking-mode and cannot find a valid condition to leave the loop i made on the read method check the code:

public static final Charset charsetUTF8 = Charset.forName("UTF-8");
public static final int BUFFER_SIZE = 1024;

public static String getUnbounded(String st, SocketAddress address) throws IOException {
    SocketChannel sc = SocketChannel.open(address);
    sc.write(charsetUTF8.encode(st));
    List<ByteBuffer> listBuffers = new ArrayList<>();
    ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
    while( sc.read(buff) > -1){
        if(buff.remaining() == 0){
            listBuffers.add(buff);  
            buff.clear();
        }
    }

    listBuffers.add(buff);
    ByteBuffer finalBuffer = ByteBuffer.allocate(BUFFER_SIZE * listBuffers.size());
    for(ByteBuffer tempBuff: listBuffers){
    finalBuffer.put(tempBuff);
        tempBuff.clear();
    }
    finalBuffer.flip();

    return charsetUTF8.decode(finalBuffer).toString();
}

Any idea on how to solve this?

Answer 1

The solution is that to get out of the loop i had to call:

sc.shutdownOutput();

Which closes the writing stream without closing the reading stream and set the sc.read(buff) to -1

Answer 2

You can't just clear() the byte buffer. You need to allocate a new one; otherwise the same buffer is being added to listBuffers repeatedly.

ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
    if(buff.remaining() == 0){
        listBuffers.add(buff);  
        buff = ByteBuffer.allocate(BUFFER_SIZE);
    }
}
if (buff.position() > 0) {
    listBuffers.add(buff);
}

Since the last buffer might not (probably will not) be full, you should calculate the finalBuffer size taking this into account.

Answer 3

The number of bytes in an HTTP response stream is not 'undefined'. See the RFC. It is defined by either:

1. EOS in the case of a connection which is closed (HTTP 1.0 or Connection: close),
2. The Content-Length header, or
3. The result of decoding the chunked-encoding format.

It is essential that it be defined in one of these ways, and maybe there are others, so that HTTP persistent connections can work, where there may be another response following this one.

I would like to know why you are implementing this at all, when the HttpURLConnection class already exists, along with various third-party HTTP clients, which already implement all this correctly, and many other things besides.