RSAES_OAEP_SHA_Decryptor MaxPlaintextLength returning zero

Question

I have to write a small authentication server that, when a client connects, sends a RSA public exponent and modulus to it, and the client returns the username and password encrypted with them. The part where I get the exponent and modulus works fine, the client receives them, and sends the encoded block back; that's not a problem. The problem is in decoding it: the RSA Decryptor always returns 0 as MaxPlaintextLength, and no matter how many different ways I try to move the data around, I can't get it to work.

I'm providing hex dumps of the different parts after the code.

// These are static in the namespace to keep the key loaded.
static CryptoPP::AutoSeededRandomPool rsaRng;
static CryptoPP::InvertibleRSAFunction rsaParameters;
static CryptoPP::RSA::PrivateKey rsaPrivateKey(rsaParameters);

// This is done when the client connects.
{
    rsaPrivateKey.GenerateRandomWithKeySize(rsaRng, 1024);

    // This is where GetPublicExponent() and GetModulus() are
    // encoded and sent to the client. They are sent correctly
    // and I receive the encrypted data in a char buffer.
}

// This runs when I receive the encrypted data, which starts
// at &dataBuffer[7] and has a length of rsaPayloadLen.
{
    int rsaPlainSize;
    byte *rsaBuffer;

    rsaBuffer = new byte[rsaPayloadLen];
    memcpy(&rsaBuffer[0], &dataBuffer[7], rsaPayloadLen);

    CryptoPP::SecByteBlock rsaCypher(rsaBuffer, rsaPayloadLen);
    CryptoPP::RSAES_OAEP_SHA_Decryptor rsaDecryptor(rsaPrivateKey);

    // At this point I inspected rsaCypher and it does contain the
    // expected binary data, and rsaCypher.size() returns 256.

    rsaPlainSize = rsaDecryptor.MaxPlaintextLength(rsaCypher.size());

    // rsaPlainSize is 0. I have no idea why!
}

Hex dumps of everything at the time of calling MaxPlaintextLength:

rsaPrivateKey.GetPublicExponent():
000000: 11                                               .

rsaPrivateKey.GetPrivateExponent():
000000: 10 7a fd fd 9e a9 72 8c c3 5d 5b 80 e8 f4 6f bc  .z....r..][...o.
000010: bc 6a 7a 51 4f 9f af d3 e4 76 a5 4a 9d fe 17 37  .jzQO....v.J...7
000020: 03 cf 82 24 33 e2 a0 d8 97 26 0a 6b ac 9d b1 de  ...$3....&.k....
000030: 39 d5 3a 93 aa 65 66 be 17 43 3c 00 20 77 68 0a  9.:..ef..C<. wh.
000040: ac 2f 77 1e b8 c4 7f 64 52 54 7c 17 54 b6 e6 a4  ./w....dRT|.T...
000050: 95 49 60 7b 7b 16 6a 41 72 54 03 a2 2d 3a 80 8b  .I`{{.jArT..-:..
000060: aa 74 fa 77 22 5d 0a d9 81 b2 b2 48 01 db 43 e8  .t.w"].....H..C.
000070: 16 1c c4 c3 a6 bf 45 7e 90 d3 6a 37 10 40 9f 71  ......E~..j7.@.q

rsaPrivateKey.GetModulus():
000000: d2 20 26 61 a6 f0 74 82 ba e6 4e ab 9a 2c 90 a6  . &a..t...N..,..
000010: 62 4d 97 8c b7 34 01 cd a0 e8 bb 77 5e 67 a7 fd  bM...4.....w^g..
000020: 70 95 bb 4d 95 89 82 c9 87 25 04 dc d8 da 9b d1  p..M.....%......
000030: 61 5e aa da bc 8c dd f7 a8 99 3d 01 9d f2 6e 89  a^........=...n.
000040: e4 75 ec 91 31 e9 86 f4 da 43 4a ca a4 66 6b 04  .u..1....CJ..fk.
000050: c2 c9 a1 18 1d fa 81 b0 6e ef a5 13 04 44 88 89  ........n....D..
000060: 42 41 be 9c 7c 77 75 96 50 07 70 ad eb 60 e5 05  BA..|wu.P.p..`..
000070: aa a8 d8 27 03 28 cf bb c7 f5 cb 0d b3 b3 96 7f  ...'.(..........

rsaPrivateKey.GetPrime1():
000000: d7 9e af ac e4 04 42 e4 58 9c 39 19 0e 56 7c ef  ......B.X.9..V|.
000010: b3 bf b6 26 73 25 d8 ab d7 5e d1 e0 56 49 ae 66  ...&s%...^..VI.f
000020: c4 d8 81 bc d0 be c2 ef f4 6a 09 72 ef 72 35 7e  .........j.r.r5~
000030: 15 f4 f9 3b f8 be f9 3a a1 0d 3e d0 eb c8 34 11  ...;...:..>...4.

rsaPrivateKey.GetPrime2():
000000: f9 7a 0e 1c 9a 1b eb d1 67 f1 e3 88 1d f3 f1 62  .z......g......b
000010: 9f a2 5c cb 49 76 de 42 25 e1 a4 de ed 50 f3 2d  ..\.Iv.B%....P.-
000020: c0 15 c3 70 b5 96 68 51 25 f7 06 24 e4 43 0d b8  ...p..hQ%..$.C..
000030: 7a c5 12 2c 7c 63 20 73 70 61 01 fe b8 b3 71 8f  z..,|c spa....q.

Plain text buffer that was encrypted:
000000: 73 74 72 69 6e 67 62 75 66 66 65 72 00 00 00 00  stringbuffer....
000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00        ..............

rsaCypher:
000000: 0e 9e bd 34 10 16 98 a5 b8 e4 0c 9b 4f 23 71 6d  ...4........O#qm
000010: af d6 e8 c1 4d 97 b9 32 cb 25 eb 01 fe 4f 5c 79  ....M..2.%...O\y
000020: 2d d8 32 c4 4f fa e9 2e 58 dd fd 37 7f 08 97 d8  -.2.O...X..7....
000030: 95 bb 6f 04 46 fa 83 77 05 01 43 75 ca be b4 4a  ..o.F..w..Cu...J
000040: 60 f9 e7 4a 91 3d bc ac fb e9 41 f3 9d b7 df d3  `..J.=....A.....
000050: a7 03 80 3a 7f 35 98 46 ca 06 b1 f3 d1 7b 56 83  ...:.5.F.....{V.
000060: 1b 00 7d 97 59 39 be 46 d5 cf 6d 2c b3 a7 8e 30  ..}.Y9.F..m,...0
000070: 39 ca ca d5 59 a2 71 43 e7 7e 75 b3 3c d6 a3 a5  9...Y.qC.~u.<...
000080: aa 89 e3 e9 32 e1 a9 c1 a5 a8 f5 66 be 7f c9 ba  ....2......f....
000090: 65 35 0f 61 a0 d4 fa c7 ac 8e 28 7c 39 26 3f 01  e5.a......(|9&?.
0000a0: 34 ad 82 69 5e c4 ab 92 48 47 42 04 02 48 79 c4  4..i^...HGB..Hy.
0000b0: 39 6e f2 2c 7c 19 71 34 36 38 1c eb c1 f2 33 f0  9n.,|.q468....3.
0000c0: 49 b9 7e bb c3 16 ed d7 f7 3e 10 a7 cc 2b 8c 31  I.~......>...+.1
0000d0: f1 17 c7 a5 49 ce dd a3 c6 e2 9c 3c 2f 37 e4 97  ....I......</7..
0000e0: ac b7 24 17 b3 f8 75 6f 2a 85 cb 23 7a e1 77 72  ..$...uo*..#z.wr
0000f0: 02 0b 90 28 9b 9b ff 5d 6f 9b 11 11 d3 8b dd 4b  ...(...]o......K

rsaCypher.size(): 256

rsaDecryptor.MaxPlaintextLength(rsaCypher.size()): 0

I'm really completely at a loss and the only reason I've spent the whole weekend fighting this is because I'm going to need CryptoPP later for Blowfish and Adler32, otherwise I would have just used another RSA library.

Answer 1

The whole problem was the key size. The doc said I should use a 1024-bit key, but the other program was expecting a 1152-byte key; apparently that change was never documented. Everything works fine after changing a single line:

rsaPrivateKey.GenerateRandomWithKeySize(rsaRng, 1152);

Leaving this answer here in case someone else ever has this problem. Check the key sizes, and don't trust the documentation!

Answer 2

RSAES_OAEP_SHA_Decryptor MaxPlaintextLength returning zero...

Your calculation of rsaPayloadLen is probably not correct. I would suggest you check the size of the ciphertext before you transmit it and after you recover it. I think its size is rsaPayloadLen in you code above.

From the comments in cryptlib.h:

//! maximum length of plaintext for a given ciphertext length
/*! \note This function returns 0 if ciphertextLength is not valid (too long or too short). */
virtual size_t MaxPlaintextLength(size_t ciphertextLength) const =0;

I cannot duplicate when I take plain text, encrypt it, and then pass ciphertext.size() into MaxPlaintextLength. I can duplicate if I pass ciphertext.size() - 1 into MaxPlaintextLength.

As a work around to get you a little further along, you might try using FixedMaxPlaintextLength(). It takes no arguments and only depends upon the public key length. You can use it to find the largest buffer based on the key. But you could fail on decryption.

You should probably show us your encryption code, too. You might not be sizing a buffer correctly.