DeflatorInputStream and DeflatorOutputStream do not reconstruct the original data

Question

I want to compress some data, so I came across the DeflatorInputStream & DeflatorOutputStream classes. However, the following example shows that I can't seem to reconstruct my original data when using these classes.

When I switch to a ZipInputStream and ZipOutputStream it does work, but since I don't need zip files per se, I thought a generic compression would be better. Mainly I'm interested in understanding why this example doesn't work.

//Create some "random" data
int bytesLength = 1024;
byte[] bytes = new byte[bytesLength];
for(int i = 0; i < bytesLength; i++) {
     bytes[i] = (byte) (i % 10);
}

//Compress the data, and write it to somewhere (a byte array for this example)
ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
DeflaterOutputStream outputStream = new DeflaterOutputStream(arrayOutputStream);
outputStream.write(bytes);

//Read and decompress the data
byte[] readBuffer = new byte[5000];
ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(arrayOutputStream.toByteArray());
DeflaterInputStream inputStream = new DeflaterInputStream(arrayInputStream);
int read = inputStream.read(readBuffer);

//Should hold the original (reconstructed) data
byte[] actuallyRead = Arrays.copyOf(readBuffer, read);

//Results differ - will print false
System.out.println(Arrays.equals(bytes, actuallyRead));

Answer 1

Blame historical precedent. On Unix, the function used to reverse a deflate is called inflate. So, unlike alot of other Java IO classes, the input and output stream pair does not have (obviously) matching names.

The DeflaterOutputStream doesn't actually allow you to reverse a deflation, instead it deflates bytes as they are passed to it from a sink to a source. DeflaterInputStream also deflates, but it performs its action as data flows from the source to the sink.

In order to read your data in uncompressed (inflated) format, you need to use an InflaterInputStream:

InflaterInputStream inputStream = new InflaterInputStream(arrayInputStream);

Also, because its possible to not get all compressed data from the stream in one read call, you need to use a loop. Something like this:

int read;
byte[] finalBuf = new byte[0], swapBuf;
byte[] readBuffer = new byte[5012];

ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(
        compressed);
InflaterInputStream inputStream = new InflaterInputStream(
        arrayInputStream);
while ((read = inputStream.read(readBuffer)) != -1) {
    System.out.println("Intermediate read: " + read);
    swapBuf = finalBuf;
    finalBuf = new byte[swapBuf.length + read];
    System.arraycopy(swapBuf, 0, finalBuf, 0, swapBuf.length);
    System.arraycopy(readBuffer, 0, finalBuf, swapBuf.length, read);
}

Finally, make sure to either flush your deflater output stream prior to retrieving the compressed bytes (or alternatively close the stream).

Answer 2

There are only 2 little changes that make your code work.

//Compress the data, and write it to somewhere (a byte array for this example)
ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
DeflaterOutputStream outputStream = new DeflaterOutputStream(arrayOutputStream);
outputStream.write(bytes);
outputStream.close();

First, you have to close() the output stream. The deflator has to make some final steps to complete his work.

InflaterInputStream inputStream = new InflaterInputStream(arrayInputStream);

If you use a Deflator InputStream, you compress the compressed data again. Replace it with an Inflator InputStream and your code will work fine.

Answer 3

OK I'm adding this response since I had a really painful implementation for consuming one API from IBM DataStage v11.7 with Java and SpringBoot (RestTemplate). I'm not sure if someone else might run into similar problems with such product or any other... But these days I had combination of issues with PKIX, API's exposed with self signed certificates and finally a deflate header. Sorry to do a big post with the three responses, but you'll be able to find similar responses for the first two issues (PKIX or endpoints with self signed certificates and RestTemplate objects). I'm adding this here since Inflater does not have many approaches yet.

Some other answers say that using a custom httpclient with resttemplate may help, or may already include a automatic "decompression" but somehow this case was not the same.

Uncompressing or inflating the response was a bit tricky and adding one extra byte at the end was the misterious "cherry on top of the cake". I managed to get the response after struggling a bit, but it was printing strange/special characters. I thought it was some encoding issue but it was actually some issue with Inflater. I had to debug to identify that the response size (lenght) was actually the same as I received in Postman. So I said, ok, lenghts are similar, so what may be happening. So, thanks to all other developers who contributed to our solution.

import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.X509Certificate;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;

import javax.net.ssl.SSLContext;

import org.apache.http.client.HttpClient;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.TrustStrategy;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.zip.DataFormatException;
import java.util.zip.Inflater;
import java.util.zip.InflaterInputStream;

    private static Map<String, String> headersMap = new HashMap<String, String>() {
        {
            put("Authorization", "Basic XYZ");
            put("Accept", "*/*");
            put("Accept-Encoding", "deflate");
            put("Connection", "keep-alive");
        }
    };

    private static final String baseUrlDesigner = "https://{HOST}/ibm/iis/api/dscdesignerapi";

    private static final String designerGetAllJobs = "?api=getJobList&hostName={HOST}&projectName={PROJECT}&compress=nowrap9";

    public static RestTemplate getRestTemplateNoSSLValidation( ) {
        TrustStrategy acceptingTrustStrategy = (X509Certificate[] chain, String authType) -> true;
        SSLContext sslContext = null;
        try {
            sslContext = org.apache.http.ssl.SSLContexts.custom()
                    .loadTrustMaterial(null, acceptingTrustStrategy)
                    .build();
        } catch (KeyManagementException e2) {
            e2.printStackTrace();
        } catch (NoSuchAlgorithmException e2) {
            e2.printStackTrace();
        } catch (KeyStoreException e2) {
            e2.printStackTrace();
        }

        SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(sslContext);

        HttpClient httpClient = HttpClients.custom()
                .setSSLSocketFactory(csf)               
                .disableContentCompression()
                .build();

        HttpComponentsClientHttpRequestFactory requestFactory =
                new HttpComponentsClientHttpRequestFactory();

        requestFactory.setHttpClient(httpClient);

        RestTemplate restTemplate = new RestTemplate(requestFactory);
        return restTemplate;
    }

    public static <T> ResponseEntity<T> sendRequest(String url, HttpMethod method, Map<String,String> headerList, Class<T> responseType) {    
        RestTemplate restTemplate = getRestTemplateNoSSLValidation(); 
        HttpHeaders headers = new HttpHeaders();        
        for (Entry<String,String> e:headerList.entrySet())          
            headers.add(e.getKey(),e.getValue());
        
        HttpEntity<String> request = new HttpEntity<String>(null, headers);     

        ResponseEntity<T> result = (ResponseEntity<T>) restTemplate.exchange(url, method, request, responseType);       
        
        return result;      
    }       

    public static String decompress(byte[] bytes) {
        Inflater inf = new Inflater(true);
        InputStream in = new InflaterInputStream(new ByteArrayInputStream(bytes), inf);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        try {
            byte[] buffer = new byte[8192];
            int len;
            while ((len = in.read(buffer)) > 0)
                baos.write(buffer, 0, len);
            baos.write(new byte[1], 0, 1); // FIX. Had to force one extra byte, otherwise the body string was not read properly :shrug:
            return new String(baos.toByteArray(), "UTF-8");
        } catch (IOException e) {
            throw new AssertionError(e);
        }
    }


    public static void main(String[] args) throws IOException {
        ResponseEntity<byte[]> responseBytes =
          sendRequest(baseUrlDesigner + designerGetAllJobs,
                HttpMethod.GET, headersMap, byte[].class);
        String respString = decompress(responseBytes.getBody());
        System.out.println("Response lenght: " + respString.length());
        System.out.println("Response body: " + respString);
    }