Strange results from bitshifting bytes and ints in Java AES implementation

Question

I'm having trouble making sense of how Java promotes bytes to ints with bitwise operations. I'm attempting to implement AES, and while my output is correct as a 2d byte array, I ultimately need to store it in a 1d int array. However, the following code changes some of the expected values

    ciphertexts[0] = ((state[0][0] & 0xFF) << 24) ^ ((state[1][0] & 0xFF) << 16)
                    ^ ((state[2][0] & 0xFF) << 8) ^ state[3][0];
    ciphertexts[1] = ((state[0][1] & 0xFF) << 24) ^ ((state[1][1] & 0xFF) << 16)
                    ^ ((state[2][1] & 0xFF) << 8) ^ state[3][1];
    ciphertexts[2] = ((state[0][2] & 0xFF) << 24) ^ ((state[1][2] & 0xFF) << 16)
                    ^ ((state[2][2] & 0xFF) << 8) ^ state[3][2];
    ciphertexts[3] = ((state[0][3] & 0xFF) << 24) ^ ((state[1][3] & 0xFF) << 16)
                    ^ ((state[2][3] & 0xFF) << 8) ^ state[3][3];    

I didn't particularly expect masking with 0xFF to help, since the mask should just return the original byte value, but then I tried this:

    int zero = ((state[0][0] & 0xFF) << 24);
    int one = ((state[0][1] & 0xFF) << 16);
    int two = ((state[0][2] & 0xFF) << 8) ;
    int three = (state[0][3] & 0xFF);
    int total = zero ^ one ^ two ^ three;

    printhex(zero);
    printhex(one);
    printhex(two);
    printhex(three);
    printhex(total);

Which gives the following output:

69000000
006A0000
0000D800
00000070
696AD870

Which is what I'm trying to do with the code above. Without the masking, the following code gives the following output:

    int zero = (state[0][0] << 24);
    int one = (state[0][1] << 16);
    int two = (state[0][2] << 8);
    int three = state[0][3];
    int total = zero ^ one ^ two ^ three;

69000000
006A0000
FFFFD800
00000070
9695D870

I also tried what seemed to me more sensible, which is masking after shifting, and got similarly messed up output:

    ciphertexts[0] = ((state[0][0] << 24) & 0xFFFFFFFF) ^ 
        ((state[1][0] << 16) & 0xFFFFFF) ^ ((state[2][0] << 8) & 0xFFFF) 
                ^ state[3][0];
    ciphertexts[1] = ((state[0][1] << 24) & 0xFFFFFFFF) ^ 
        ((state[1][1] << 16) & 0xFFFFFF) ^ ((state[2][1] << 8) & 0xFFFF) 
                ^ state[3][1];
    ciphertexts[2] = ((state[0][2] << 24) & 0xFFFFFFFF) ^ 
        ((state[1][2] << 16) & 0xFFFFFF) ^ ((state[2][2] << 8) & 0xFFFF)
                ^ state[3][2];
    ciphertexts[3] = ((state[0][3] << 24) & 0xFFFFFFFF) ^ 
        ((state[1][3] << 16) & 0xFFFFFF) ^ ((state[2][3] << 8) & 0xFFFF)
                ^ state[3][3];

Where "messed up" means:

ciphertext at round 9 is 963b1fd86a7b04302732488070b4c55a 

instead of:

69C4E0D86A7B0430D8CDB78070B4C55A

So my questions are how do I neatly or bytes together into an int, and what is actually going on with the masking and shifting. I looked at other answers and can't figure out why they're not working in this case. Thanks!

Answer 1

That´s the cruelty of a language lacking unsigned
(one can get the same result in C if he/she use a signed char, ie. signed byte)
Let´s ignore shift´s , only concentrate at the assignment and &.
Example value here 0xfe instead of 0xd8
(the problem will happen with each value between 0x80 and 0xff)

With problem, java:

byte a = 0xfe;
int i = a;

With problem, C:

signed char a = 0xfe;
int i = a;

What does happen: A byte can hold value between -128 and +127.
0xfe maps to a negative number (2-complement): -2
...and so, i get the value -2 in i, and i is not 8bit, but 32bit long.
According to the rules of the 2-complement, this gives 0xfffffffe
(http://en.wikipedia.org/wiki/Two%27s_complement)

So, what does & change, because masking 0xfe first with 0xff
shouldn´t change the value?
Yes, but: As & is a "calculation" like + - ...
the value gets expanded first to 32bit
(because more suited for the processor´s ALU)

That´s more likely to be known by C/Asm programmers,
but as you see, it´s relevant in Java too.
(if nessecary for an assignment to an smaller variable than 32bit,
it will be shortened again after calculation)

Ie. first, -2=0xfe becomes 32bit -2=0xfffffffe,
then masking results in a 0xfe again (already 32bit)...
which is assigned to i.

Answer 2

Your value of state[0][2] is a byte 0xD8. This has the most significant bit set to 1: in binary: 1101 1000. Before the shift operation << is applied, the byte is converted to an int. Java doesn't care that byte is unsigned, it is treated as a signed byte. So the byte's most significant bit is filled all the way to the int's most significant bit.

In short: With bytes you need the mask with 0xFF as this masks the filled in bits away in the already converted int.