Passing gzencoded data to mcrypt_encrypt, can't unzip after decryption

Question

To keep this short, are there known issues when passing results of gzencode (or other non-text data) to mcrypt_encrypt functions?

Details:

Basically I have an issue where encryption/decryption is working for plain text but I get errors unzipping if I pass compressed data to the encryption function, then decrypt and unzip.

So in PHP, I'm passing results of gzencode() to the encrypt function. Then I base64 encode for showing results on a web service web page. Then in a Java app I'm decoding base64, decrypting, and unzipping using GZIPInputStream. I get errors in the last step.

But everything works fine if I skip the compression step (just pass plain text to the encrypt function). Everything also works fine if I skip encryption and just do compression. So those functions seem to work fine on both PHP and Java sides if I don't combine them.

public static function encrypt($str,$key,$iv) {
    $str=Crypto2::pkcs5Pad($str,mcrypt_get_block_size(MCRYPT_RIJNDAEL_128, MCRYPT_MODE_CBC));
    $encrypted=mcrypt_encrypt(MCRYPT_RIJNDAEL_128,$key,$str,MCRYPT_MODE_CBC,$iv);
    return $encrypted;
}

public static function pkcs5Pad ($text, $blocksize) {
    $pad = $blocksize - (strlen($text) % $blocksize);
    $padded=$text . str_repeat(chr($pad), $pad);
    return $padded;
} 

Java functions:

public static byte[] decrypt(byte[] inputbuffer) throws Exception {
    Key key = new SecretKeySpec(keybyte, "AES");
    IvParameterSpec ivSpec = new IvParameterSpec(iv);
    Cipher c = Cipher.getInstance("AES/CBC/PKCS5Padding");
    c.init(Cipher.DECRYPT_MODE, key, ivSpec);

    c.getBlockSize();
    System.out.println("Block size="+c.getBlockSize());

    int outlen = c.getOutputSize(inputbuffer.length);
    System.out.println("Output length will be:"+outlen);
    byte[] result=c.doFinal(inputbuffer);
    return result;
}

  public static byte[] decodeBase64(String data) throws IOException{ 
    BASE64Decoder decoder = new BASE64Decoder(); 
    byte[] decodedBytes = decoder.decodeBuffer(data);

    return decodedBytes;
  }


   public static void unzipPrint(byte[] data) throws Exception{
    InputStream is=new GZIPInputStream(new ByteArrayInputStream(data));
    int ch2;
    while((ch2=is.read())!=-1) {
        System.out.print((char)ch2);
    }
   }

So if I do this in PHP: base64_encode(encrypt(gzencode($plain_text)));

and this in Java

unzipPrint(decrypt(decodeBase64(data)));

I get the dreaded: "java.util.zip.ZipException: oversubscribed dynamic bit lengths tree" during the unzip phase.

Again, if I skip the compression/decompression steps at both ends everything is fine. And if I skip encryption at both ends then compression/decompression works fine.

EDIT: Ok weird, but after checking byte by byte the resulting byte array of the compressed data (after decoding base64 and decryption), I found a SINGLE byte that was off (compared to the original PHP byte array) by a value of 1. It was byte number 14 (index 13 in Java) and it had a value of 110 instead of 111. I have absolutely no idea how this could be the case.

So if I change that single byte to from 110 to 111, then I can successfuly use GZIPOutputStream to uncompress the data.

So I know what is wrong but not why.

EDIT 2: This is SOLVED ->Thanks to comment by Owlstead I double checked the IV values and found that there was minor descrepancy between the php and java code. How this can lead to only one byte of difference in the resulting decrypted data I have no idea.

That was one wasted day over a single 0x13 instead of 0x12 in my IV.

Answer 1

You should check the IV, as it may change only the first block of cipher text, which holds the ZIP header.

(to close the question, glad you got it solved, any day that you solve an issue is not one wasted in my opinion :) )