CryptDecryptMessage failing

Question

I have been trying to write some code to use the Microsoft CryptoAPI. The goal is very simple: encrypt and then decrypt a string. I seem to have it almost working, but the very last call to CryptDecryptMessage fails.

So, I am able to successfully encrypt my string. When I then take the encrypted binary string and try to decrypt it, everything works except the last call to CryptDecryptMessage . The decryption code I am using is below.

As the code stands, the first call to CryptDecryptMessage, to get the required size of the output buffer, succeeds, BUT it always returns a size equal to the cleartext string size plus 6. I then size up my output buffer, and the next call to CryptDecryptMessage fails (return value is zero, dwSizeRequired gets set to zero, nothing is put in the output buffer, and GetLastError returns NTE_BAD_KEY).

On the other hand, if I uncomment the #define TESTING_INCORRECT line, the behaviour is slightly different. The first call to CryptDecryptMessage succeeds. Its return value is zero, but GetLastError returns ERROR_MORE_DATA, which one would expect, and most importantly dwSizeRequired is set with the size of the original cleartext string that was encoded. I then allocate and call CryptDecryptMessage the second time, it fails again as above.

Lastly, I do have the private certificate key in my keystore.

Does anyone have any idea what is wrong here????? Thanks.

void decrypt(std::string& sClearTextString , const std::vector<T_Byte>& sEncryptedBinaryString, const std::string& sKey)
{
    BOOL bResult;

    HCRYPTPROV hProv;
    bResult = CryptAcquireContext( &hProv, NULL, MS_DEF_PROV, PROV_RSA_FULL, 0); 
    CHECK_CRYPT_ERR( !bResult , "Unable to find crypto context");

    HCERTSTORE hStore(NULL);
    const char* const pcKey = sKey.empty()? "MY" : sKey.c_str();
    hStore = CertOpenSystemStore( 0, pcKey);
    CHECK_CRYPT_ERR( hStore==NULL, "unable to open certificate store.");

    HCERTSTORE CertStoreArray[] = {hStore};

    CRYPT_DECRYPT_MESSAGE_PARA  DecryptParams;
    DWORD  DecryptParamsSize = sizeof(DecryptParams);
    memset(&DecryptParams, 0, DecryptParamsSize);
    DecryptParams.cbSize = DecryptParamsSize;
    DecryptParams.dwMsgAndCertEncodingType = (PKCS_7_ASN_ENCODING | X509_ASN_ENCODING);
    DecryptParams.cCertStore = sizeof(CertStoreArray)/sizeof(HCERTSTORE);
    DecryptParams.rghCertStore = CertStoreArray;

    const BYTE* const pbContent = &sEncryptedBinaryString[0];
    DWORD dwSize = STATICCAST<DWORD>(sEncryptedBinaryString.size());
    DWORD dwSizeRequired = 0;

    BYTE* pbOutBuffer = NULL;
//#define TESTING_INCORRECT
#ifdef TESTING_INCORRECT
    std::string sDummyBuffer(2,'\0');
    pbOutBuffer = (BYTE*)(&sDummyBuffer[0]);
#endif

    // Get required buffer size.
    bResult = CryptDecryptMessage( &DecryptParams, pbContent, dwSize, pbOutBuffer, &dwSizeRequired, NULL);
    CHECK_CRYPT_ERR( !bResult && ERROR_MORE_DATA != GetLastError() , "Unable to get buffer length");

    //Allocate buffer
    sClearTextString.clear();
    sClearTextString.resize(dwSizeRequired+1,0);
    pbOutBuffer = (BYTE*)(&sClearTextString[0]);

    //Now actually decryt
    bResult = CryptDecryptMessage( &DecryptParams, pbContent, dwSize, pbOutBuffer, &dwSizeRequired, NULL);
    CHECK_CRYPT_ERR( !bResult , "Unable to decrypt");

    CertCloseStore(hStore, CERT_CLOSE_STORE_CHECK_FLAG);
    CryptReleaseContext(hProv, 0);
}

Answer 1

I would investigate the keys being used. The return code of 2148073475 actually translates to 0x80090003, which is NTE_BAD_KEY. GetLastError of 234 is ERR_MORE_DATA, which indicates an undersized buffer.

I think the problem lies in the expectation that the first call to CryptDecryptMessage will receive NULL as the pointer to pbDecrypted because its a sizing call, but as its non-null with the comment removed, it gives the 234 result thinking its trying to decrypt something that's already been sized.

If the presumption that the receipt buffer on the sizing (first) call should be null is correct, then that leads me to suggest that the key(s) may be suspect in some way per the NTE_BAD_KEY error code.

Hope that stirs some helpful thought for you.