RSAES_OAEP_SHA_Decryptor MaxPlaintextLength returning zero...
Your calculation of rsaPayloadLen
is probably not correct. I would suggest you check the size of the ciphertext before you transmit it and after you recover it. I think its size is rsaPayloadLen
in you code above.
From the comments in cryptlib.h
:
//! maximum length of plaintext for a given ciphertext length
/*! \note This function returns 0 if ciphertextLength is not valid (too long or too short). */
virtual size_t MaxPlaintextLength(size_t ciphertextLength) const =0;
I cannot duplicate when I take plain text, encrypt it, and then pass ciphertext.size()
into MaxPlaintextLength
. I can duplicate if I pass ciphertext.size() - 1
into MaxPlaintextLength
.
As a work around to get you a little further along, you might try using FixedMaxPlaintextLength()
. It takes no arguments and only depends upon the public key length. You can use it to find the largest buffer based on the key. But you could fail on decryption.
You should probably show us your encryption code, too. You might not be sizing a buffer correctly.