Own RC4 algorithm giving wrong output

Question

I've written the following implementation of the RC4 algorithm, where key is a RC4_KEY struct as given in the OpenSSL library. m_key is a QByteArray containing the set key. For this test, I have been using "teste" as key. As you can see, I have two more QByteArrays, one of them holding the original (input) data and the other one the encrypted (output) data.

void rc4SetKey() {
    for (int i = 0; i < 256; ++i) {
        key.data[i] = i;
    }
    for (int i = 0; i < 256; ++i) {
        int j = (j + key.data[i] + m_key[i % m_key.length()]) % 256;
        std::swap(key.data[i], key.data[j]);
    }
}

void rc4Encrypt(QByteArray &in, QByteArray &out) {
    out.clear();
    out.resize(in.length());
    for (int n = 0; n < in.length(); ++n) {
        int i = (i + 1) % 256;
        int j = (j + key.data[i]) % 256;
        std::swap(key.data[i], key.data[j]);
        int rand = key.data[(key.data[i] + key.data[j]) % 256];
        out[n] = rand ^ in[n];
    }
}

For testing purposes, I am using a text file with the following data (as hex):

31 32 33 34 35 36 37 38 38 39 31 30 0a

Using an online tool or the OpenSSL function, I get the following output (key: "teste"):

6a 9d ae b6 17 61 7b 71 5f f7 46 f0 ab

Using my own implementation, however, I get the following:

52 ec c2 b1 3d ca 6b 55 50 54 30 e7 ed

I have spent quite some time now and looked at various pseudo codes or implementations, but I am still not able to figure out where I went wrong.

Answer 1

This expression (in both the set-key and encrypt functions) is illegal (compiling with all warnings turned on should have pointed it out):

 int j = (j + ...

In addition to being nonsense (trying to reference the value of a variable in its initialization), it doesn't match the algorithm definition - j is supposed to be declared outside the loop and not reset on each iteration.

The following should fix the key-set function. The fix for the encrypt function will be nearly the same (it also needs i to be fixed in the same way).

int j = 0;
for (int i = 0; i < 256; ++i) {
    j = (j + key.data[i] + m_key[i % m_key.length()]) % 256;
    std::swap(key.data[i], key.data[j]);
}