AES_ctr128_encrypt on Solaris doesnt decrypt same as other OS's

Question

I've compiled the libcrypto.so shared library from openssl-1.0.1g for various platforms. Using the shared library, my app is able to encrypt/decrypt data between any of the following:

• Linux i386
• Linux x86_64
• Linux PPC
• AIX RISC

These can decrypt their own data, but NOT each other, nor any of the above:

• Solaris i386
• Solaris SPARC

I first thought this was an endian issue, butAIX/RISC and Linux/PPC are BE, and they seem to play well wit the Linux i386 and x86_64 (LE).

The Solaris systems are:

SunOS pluto 5.9 Generic_Virtual sun4u sparc SUNW,SPARC-Enterprise-T5120
SunOS nemo 5.9 Generic_118559-11 i86pc i386 i86pc

The libcrypto.so was compiled with:

#[openssl-1.0.1g] ./Configure solaris-sparcv9-cc shared 
#[openssl-1.0.1g] ./Configure solaris-x86-cc shared

NOTE that the Solaris systems both use "/opt/SUNWspro/bin/cc" as the compiler, and all others use "gcc".

I have verified that I am always starting with the same IV/counter and symkey when encrypting and decrypting. The encryption routines are well embededed in my programs, but this is basically all that I'm doing:

uchar ivec[AES_BLOCK_SIZE], ecount[AES_BLOCK_SIZE], symkey[16];
uint num = 0, bits = 128;
uchar *buf, *ebuf; /* these are malloc'd and buf set to data to encrypt/decrypt */
...
memcpy(symkey, MYTESTKEY, 16);
memset(ecount, 0, AES_BLOCK_SIZE);
memset(ivec, 0, AES_BLOCK_SIZE);
memcpy(ivec, SOMEDATA, 8); /* from RAND_seed() and RAND_bytes() */
AES_set_encrypt_key(symkey, bits, &outkey);
...
AES_ctr128_encrypt(inp, outp, bytes, key, ivec, ecount, &num);

Again, this always works when encrypting/decrypting on the same system, only decodes to junk when one of the systems is Solaris. I'm using very old Solaris OS and boxes, however, since I have to maintain backward compatibility. Thanks in advance!

Answer 1

Just to update, I converted to EVP but no difference. I recompiled libcrypto.so with GCC also with no difference (using solaris64-sparcv9-gcc) - until I removed the link to cc from /usr/bin. Then I recompiled using GCC again and this time it worked. I didn't have time to investigate further, but it implies that some of the objects were still being compiled with cc, even after specifying *-gcc. Unfortunately there's no GCC available for our old Solaris-x86 V9 system, so we won't be supporting V9 on intel anymore.

Answer 2

Please use the higher end OpenSSL EVP functionality.

Check for instance this message by one of the core devs:

There is no guarantee that the low level cipher context (or indeed any similar context) will be the same across platforms. Endian differences, padding and platform specific optimisations can result in different data at the machine level. The results of using contexts on the appropriate platform should obviously be consistent though.