HttpClient Connection closed in between while data is transferring

Question

while getting the xml from server, i am facing an issue, where the the response xml provided by HttpClient GET method is partial. Whereas the xml provide using browser is full as expected.

The code for the same is

        InputStream response = null;
        HttpClient client = new HttpClient();
        method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));

        int statusCode = client.executeMethod(method);

        if (statusCode != HttpStatus.SC_OK) {
        System.err.println("Method failed: " + method.getStatusLine());
        }

        response = method.getResponseBodyAsStream();

following is the output of httpclient debug logs

        16:07:24(GetMethod.java:89) DEBUG org.apache.commons.httpclient.methods.GetMethod - enter GetMethod(String)
        16:07:26(DefaultHttpParams.java:150) DEBUG org.apache.commons.httpclient.params.DefaultHttpParams - Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@578cef18
        16:07:29(HttpClient.java:322) DEBUG org.apache.commons.httpclient.HttpClient - enter HttpClient.executeMethod(HttpMethod)
        16:07:29(HttpClient.java:373) DEBUG org.apache.commons.httpclient.HttpClient - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
        16:07:29(HttpMethodDirector.java:378) DEBUG org.apache.commons.httpclient.HttpMethodDirector - Attempt number 1 to process request
        16:07:29(HttpConnection.java:684) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.open()
        16:07:29(HttpConnection.java:691) DEBUG org.apache.commons.httpclient.HttpConnection - Open connection to mysite:80
        16:07:29(HttpMethodBase.java:976) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.execute(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1916) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequest(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:2071) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1379) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.generateRequestLine(HttpConnection, String, String, String, String)
        16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "GET /mypath HTTP/1.1[\r][\n]"
        16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
        16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
        16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
        16:07:29(HttpMethodBase.java:2034) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
        16:07:29(HttpMethodBase.java:1300) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1325) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1211) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1235) DEBUG org.apache.commons.httpclient.HttpMethodBase - Adding Host request header
        16:07:29(HttpMethodBase.java:1163) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
        16:07:29(HttpState.java:172) DEBUG org.apache.commons.httpclient.HttpState - enter HttpState.getCookies()
        16:07:29(CookieSpecBase.java:548) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.match(String, int, String, boolean, Cookie[])
        16:07:29(HttpMethodBase.java:1262) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
        16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "User-Agent: Jakarta Commons-HttpClient/3.0.1[\r][\n]"
        16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
        16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
        16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
        16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "Host: mysite[\r][\n]"
        16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
        16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
        16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
        16:07:29(HttpConnection.java:1000) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.writeLine()
        16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
        16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
        16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "[\r][\n]"
        16:07:29(HttpConnection.java:825) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.flushRequestOutputStream()
        16:07:29(HttpMethodBase.java:1585) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponse(HttpState, HttpConnection)
        16:07:29(HttpMethodBase.java:1823) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readStatusLine(HttpState, HttpConnection)
        16:07:29(HttpConnection.java:1112) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.readLine()
        16:07:29(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:07:29(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:07:29(Wire.java:69) DEBUG httpclient.wire.header - << "HTTP/1.0 200 OK[\r][\n]"
        16:09:31(HttpMethodBase.java:1785) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
        16:09:31(HttpConnection.java:856) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.getResponseInputStream()
        16:09:31(HttpParser.java:159) DEBUG org.apache.commons.httpclient.HttpParser - enter HeaderParser.parseHeaders(InputStream, String)
        16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:09:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:09:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
        16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-DGKeepAlive: on[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Date: Thu, 10:37:29 GMT[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Server: Apache[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Set-Cookie: JSESSIONID=1DA9A4C37BD6883B837BA8725E3CCC5D; Path=/mypath[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Vary: Accept-Encoding[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Content-Type: text/xml;charset=ISO-8859-1[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-Cache: MISS from ajay[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-Cache-Lookup: MISS from ajay:3128[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Via: 1.0 ajay:3128 (squid/2.6.STABLE21)[\r][\n]"
        16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Proxy-Connection: close[\r][\n]"
        16:11:31(HttpMethodBase.java:1462) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
        16:11:31(CookieSpecBase.java:242) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.parse(String, port, path, boolean, String)
        16:11:31(CookieSpecBase.java:112) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.parse(String, port, path, boolean, Header)
        16:11:31(HeaderElement.java:193) DEBUG org.apache.commons.httpclient.HeaderElement - enter HeaderElement.parseElements(char[])
        16:11:31(Cookie.java:108) DEBUG org.apache.commons.httpclient.Cookie - enter Cookie(String, String, String, String, Date, boolean)
        16:11:31(RFC2109Spec.java:130) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter RFC2109Spec.validate(String, int, String, boolean, Cookie)
        16:11:31(CookieSpecBase.java:368) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.validate(String, port, path, boolean, Cookie)
        16:11:31(HttpState.java:124) DEBUG org.apache.commons.httpclient.HttpState - enter HttpState.addCookie(Cookie)
        16:11:31(RFC2109Spec.java:243) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter RFC2109Spec.formatCookie(Cookie)
        16:11:31(HttpMethodBase.java:1506) DEBUG org.apache.commons.httpclient.HttpMethodBase - Cookie accepted: "$Version=0; JSESSIONID=1DA9A4C37BD6883B837BA8725E3CCC5D; $Path=/mypath"
        16:11:31(HttpMethodBase.java:1636) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
        16:11:31(HttpMethodBase.java:1671) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseBody(HttpConnection)
        16:11:31(HttpConnection.java:856) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.getResponseInputStream()
        16:11:31(HttpMethodBase.java:2140) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.canResponseHaveBody(int)
        16:11:31(Wire.java:83) DEBUG httpclient.wire.content - <<"#######INCOMPLETE XML#########"
        16:12:12(HttpMethodBase.java:921) DEBUG org.apache.commons.httpclient.HttpMethodBase - Resorting to protocol version default close connection policy
        16:12:12(HttpMethodBase.java:929) DEBUG org.apache.commons.httpclient.HttpMethodBase - Should close connection, using HTTP/1.0
        16:12:12(HttpConnection.java:1147) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.close()
        16:12:12(HttpConnection.java:1214) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.closeSockedAndStreams()
        16:12:12(HttpConnection.java:1173) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.releaseConnection()
        16:12:12(HttpConnection.java:1177) DEBUG org.apache.commons.httpclient.HttpConnection - Releasing connection back to connection manager.

When the same url is fetched through browser, it works. Headers of request are following Browser Response Headersview source

        Content-Encoding    gzip
        Content-Type    text/xml;charset=ISO-8859-1
        Date    Thu, 20 Jun 2013 11:24:54 GMT
        Proxy-Connection    close
        Server  Apache
        Vary    Accept-Encoding
        Via 1.0 ajay:3128 (squid/2.6.STABLE21)
        X-Cache MISS from ajay
        X-Cache-Lookup  MISS from ajay:3128

Browser Request Headersview source

        Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
        Accept-Encoding gzip, deflate
        Accept-Language en-US,en;q=0.5
        Connection  keep-alive
        Cookie  JSESSIONID=6265E9445BB5CB5EF2594A9BEC19B119
        Host    mysite
        User-Agent  Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0

Can anyone explain where is the issue, in the httpclient calling code or some other at server side and how to correct that? Thanks in advance.

The exception is thrown while parsing the response

       DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(inputStreamOfResponse);

exception is

    [Fatal Error] :2:1: XML document structures must start and end within the same entity.
    org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 1; XML document structures must start and end within the same entity.

Answer 1

finally solved the issue, setting the following header worked for me.

method.setRequestHeader("Accept-Encoding", "gzip, deflate");

and

postResponse = IOUtils.toString(new GZIPInputStream(method.getResponseBodyAsStream()));