Why do I get empty request from the Jakarta Commons HttpClient?

Question

I have a problem with the Jakarta Commons HttpClient. Before my self-written HttpServer gets the real request there is one request which is completely empty. That's the first problem. The first problem is solved. It was caused by an unnecessary URLConnection! The second problem is, sometimes the request data ends after the third or fourth line of the http request:

POST / HTTP/1.1
User-Agent: Jakarta Commons-HttpClient/3.1
Host: 127.0.0.1:4232

For debugging I am using the Axis TCPMonitor. There every things is fine but the empty request.

How I process the stream:

StringBuffer requestBuffer = new StringBuffer();

InputStreamReader is = new InputStreamReader(socket.getInputStream(), "UTF-8");

int byteIn = -1;
do {
    byteIn = is.read();
    if (byteIn > 0) {
        requestBuffer.append((char) byteIn);
    }
} while (byteIn != -1 && is.ready());

String requestData = requestBuffer.toString();

Found a new way to process the stream. I read all header parameters and use the 'content-length' for reading the post data.

InputStream is = mySocket.getInputStream();
if (is == null) {
    return;
}
BufferedReader in = new BufferedReader(new InputStreamReader(is, "UTF-8"));

// Read the request line
// ...
// ...

// Parse the header
Properties header = new Properties();
if (st.hasMoreTokens()) {
    String line = in.readLine();
    while (line != null && line.trim().length() > 0) {
        int p = line.indexOf(':');
        header.put(line.substring(0, p).trim().toLowerCase(), line.substring(p + 1).trim());
        line = in.readLine();
    }
}

// If the method is POST, there may be parameters
// in data section, too, read it:
String postLine = "";
if (method.equalsIgnoreCase("POST")) {
    long size = 0x7FFFFFFFFFFFFFFFl;
    String contentLength = header.getProperty("content-length");
    if (contentLength != null) {
        try {
            size = Integer.parseInt(contentLength);
        } catch (NumberFormatException ex) {
        }
    }
    postLine = "";
    char buf[] = new char[512];
    int read = in.read(buf);
    while (read >= 0 && size > 0 && !postLine.endsWith("\r\n")) {
        size -= read;
        postLine += String.valueOf(buf, 0, read);
        if (size > 0) {
            read = in.read(buf);
        }
    }
    postLine = postLine.trim();
    decodeParms(postLine, parms);
}

How I send the request:

client.getParams().setSoTimeout(30000);

method = new PostMethod(url.getPath());
method.getParams().setContentCharset("utf-8");
method.setRequestHeader("Content-Type", "application/xml; charset=utf-8");
method.addRequestHeader("Connection", "close");
method.setFollowRedirects(false);

byte[] requestXml = getRequestXml();

method.setRequestEntity(new InputStreamRequestEntity(new ByteArrayInputStream(requestXml)));

client.executeMethod(method);

int statusCode = method.getStatusCode();

Have anyone of you an idea how to solve these problems the problem?

Alex

Answer 1

It might be to do with the second condition in your while loop, the isReady() method might return false when the next read might block - but you don't really care if it blocks or not, so we can simply remove it (you can read more here: http://java.sun.com/j2se/1.5.0/docs/api/java/io/InputStreamReader.html#ready%28%29 ). Try changing to this:

byte[] buf = new byte[500];
while((is.read(buf))>-1){
  requestBuffer.append(new String(buf).trim());
  buf = new byte[500];
}

Now you should get the whole request.

Answer 2

I don't know about the first problem, but I think your second problem is due to this:

} while (byteIn != -1 && is.ready());

If the sender is not quick enough sending data, the receiver may call is.ready() before the next packet has been sent. This will cause is.ready() to return false which will cause the loop to stop.

The minimal fix is to change that line to:

} while (byteIn != -1);

EDIT

But really, you need to rewrite the method along the lines of @simonlord's answer. It is a really bad idea to read an unbuffered stream one byte at a time. You end up doing a system call for each read call, which is horribly inefficient.

EDIT 2

The reason that removing is.ready() caused delays is because you were not paying proper attention to the HTTP protocol. The problem was the HttpClient code was keeping the request side of the TCP connection open to allow the connection to be reused. The simple (but sub-optimal) solution would have been to configure HttpClient to close the request side of the connection. Your code would have then seen the EOF straight away. What you actually did was another solution.

Frankly you should not even be trying to implement the server-side HTTP protocol unless you are prepared to understand the entire HTTP specification in depth and implement it faithfully. The chances are that an existing implementation will be faster and more reliable than anything that you can knock together. The problem with implementing a subset of the spec is that your server may need to talk to a real browser that uses parts of the spec that you haven't bothered to implement / test.