HttpURLConnection does not read the whole response

Question

I use HttpURLConnection to do HTTP POST but I dont always get back the full response. I wanted to debug the problem, but when I step through each line it worked. I thought it must be a timing issue so I added Thread.sleep and it really made my code work, but this is only a temporary workaround. I wonder why is this happening and how to solve. Here is my code:

public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException {

    URL u = new URL(url);
    URLConnection c = u.openConnection();
    InputStream in = null;
    String mediaType = null;
    if (c instanceof HttpURLConnection) {

        //c.setConnectTimeout(1000000);
        //c.setReadTimeout(1000000);

        HttpURLConnection h = (HttpURLConnection)c;
        h.setRequestMethod("POST");
        //h.setChunkedStreamingMode(-1);
        setAccept(h, expectedMimeType);
        h.setRequestProperty("Content-Type", inputMimeType);

        for(String key: httpHeaders.keySet()) {
            h.setRequestProperty(key, httpHeaders.get(key));

            if (logger.isDebugEnabled()) {
                logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key));
            }

        }

        h.setDoOutput(true);
        h.connect();

        OutputStream out = h.getOutputStream();

        out.write(input.getBytes());

        out.close();

        mediaType = h.getContentType();

        logger.debug(" ------------------ sleep ------------------ START");
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        logger.debug(" ------------------ sleep ------------------ END");

        if (h.getResponseCode() < 400) {
            in = h.getInputStream();
        } else {
            in = h.getErrorStream();
        }
    }
    return in;

}

later I do the following to read the input stream

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        while (is.available() > 0) {
            bos.write(is.read());
        }
        is.close();

        //is.read(bytes);
        if (logger.isDebugEnabled()) {
            logger.debug(" Response lenght is : " + is.available());
            //logger.debug("RAW response is " + new String(bytes));
            logger.debug("RAW response is " + new String(bos.toByteArray()));
        }

It genearates the following HTTP headers

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: keep-alive
Content-Length: 1107

In other posts it was suggested to turn off keep-alive by using the

http.keepAlive=false

system property, I tried that and the headers changed to

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: close
Content-Length: 1107

the Connection header is "close" but I still cannot read the whole response. Any idea what do I do wrong?

Answer 1

I think your problem is in this line:

while (is.available() > 0) {

According to the javadoc, available does not block and wait until all data is available, so you might get the first packet and then it will return false. The proper way to read from an InputStream is like this:

int len;
byte[] buffer = new byte[4096];
while (-1 != (len = in.read(buffer))) {
  bos.write(buffer, 0, len);
}

Read will return -1 when there nothing left in the inputstream or the connection is closed, and it will block and wait for the network while doing so. Reading arrays is also much more performant than using single bytes.

Answer 2

Maybe I missed it, but what's the datatype of "input" in your code? Something that's strange about InputStreams in general is that the read( ... ) methods tend to block until data is available, then return only that data. You'll actually need to keep reading from your InputStream and appending to a ByteArrayInputStream or some other structure until you explicitly force a EOFException.

Answer 3

If you are reading the whole Message at once you can compare the isr.available() to the expected content lenght. This is how I did it:

public byte[] readData(HttpURLConnection conn)
        throws IOException, InterruptedException {
    String _connlen = conn.getHeaderField("Content-Length");
    int connlen = Integer.parseInt(_connlen);
    InputStream isr = null;
    byte[] bytes = new byte[connlen];

    try {
        isr = conn.getInputStream();

        //security count that it doesn't begin to hang
        int maxcounter = 0;
        //wait till all data is avalibal, max 5sec
        while((isr.available() != connlen) && (maxcounter  < 5000)){
            Thread.sleep(1);
            maxcounter++;
        }
        //Throw if not all data could be read
        if(maxcounter >= 5000)
            throw new IllegalAccessError(); 

        //read the data         
        if(isr.read(bytes, 0, connlen) < 0)
            throw new IllegalAccessError();     


    } finally {
        if (isr != null)
            isr.close();
    }

    return bytes;
}