A bit confused about HttpClient[Java] handling gzip responses

Question

My application makes a http request to some api service, that service returns a gzipped response. How can I make sure that the response is indeed in gzip format? I'm confused at why after making the request I didn't have to decompress it.

Below is my code:

public static String streamToString(InputStream stream) {
    BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
    StringBuilder sb = new StringBuilder();
    String line;

    try {
        while ((line = reader.readLine()) != null) {
            sb.append(line).append("\n");
        }
    } catch (IOException e) {
        logger.error("Error while streaming to string: {}", e);
    } finally {
        try { stream.close(); } catch (IOException e) { }
    }

    return sb.toString();
}

public static String getResultFromHttpRequest(String url) throws IOException { // add retries, catch all exceptions
    HttpClient httpclient = new DefaultHttpClient();
    HttpGet httpGet;
    HttpResponse httpResponse;
    InputStream stream;

    try {
        httpGet = new HttpGet(url);
        httpGet.setHeader("Content-Encoding", "gzip, deflate");
        httpResponse = httpclient.execute(httpGet);
        logger.info(httpResponse.getEntity().getContentEncoding());
        logger.info(httpResponse.getEntity().getContent());
        if (httpResponse.getStatusLine().getStatusCode() == 200) {
            stream = httpResponse.getEntity().getContent();
            return streamToString(stream);
        }
    } catch (IllegalStateException e) {
        logger.error("Error while trying to access: " + url, e);
    }

    return "";
}

Maybe it is decompressing it automatically, but I would like to see some indication of that at least.

Answer 1

Hi I am late but this answer might by used who is facing same issue. By default content is decompressed in the response. So, you have to disable the default compression using following code:

CloseableHttpClient client = HttpClients.custom()
    .disableContentCompression()
    .build();

HttpGet request = new HttpGet(urlSring);
request.setHeader(HttpHeaders.ACCEPT_ENCODING, "gzip");

CloseableHttpResponse response = client.execute(request, context);
HttpEntity entity = response.getEntity();
Header contentEncodingHeader = entity.getContentEncoding();

if (contentEncodingHeader != null) {
    HeaderElement[] encodings =contentEncodingHeader.getElements();
    for (int i = 0; i < encodings.length; i++) {
        if (encodings[i].getName().equalsIgnoreCase("gzip")) {
            entity = new GzipDecompressingEntity(entity);
            break;
        }
    }
}

String output = EntityUtils.toString(entity, Charset.forName("UTF-8").name());

Answer 2

I think you want to use DecompressingHttpClient (or the new HttpClientBuilder - which adds that header by default, don't call disableContentCompression - I don't think DefaultHttpClient supports compression by default). The client needs to send an Accept-Encoding header, Content-Encoding comes from the server response.

Answer 3

httpResponse.getEntity().getContentEncoding()

You can find out whether or not an entity requires decompression by examining its Content-Encoding header. This header will be rewritten (or removed) in case of automatic content decompression.

Answer 4

Since 4.1, Apache HttpClients handles request and response compression. You can check the example in another answer here.

Still in case you want to check whether the response was compressed or not. You can print the class of the entity.

HttpResponse httpResponse = client.execute(request);
HttpEntity httpEntity = httpResponse.getEntity();
System.out.println(httpEntity.getClass().getName());

In case of gzip the output will be org.apache.http.client.entity.GzipDecompressingEntity & for deflate its org.apache.http.client.entity.DecompressingEntity