This expression (in both the set-key and encrypt functions) is illegal (compiling with all warnings turned on should have pointed it out):
int j = (j + ...
In addition to being nonsense (trying to reference the value of a variable in its initialization), it doesn't match the algorithm definition - j
is supposed to be declared outside the loop and not reset on each iteration.
The following should fix the key-set function. The fix for the encrypt function will be nearly the same (it also needs i
to be fixed in the same way).
int j = 0;
for (int i = 0; i < 256; ++i) {
j = (j + key.data[i] + m_key[i % m_key.length()]) % 256;
std::swap(key.data[i], key.data[j]);
}