Windows CryptoAPI: CryptSignHash with CALG_SHA_256 and private key from MY keystore

Question

I am trying to generate digital signatures on Windows (from XP SP3, but currently testing with Windows 7) with CryptoAPI that will be compatible with the following openssl commands:

openssl dgst -sha256 -sign <parameters> (for signing)
openssl dgst -sha256 -verify <parameters> (for validation)

I want to use a private key from the Windows "MY" keystore for signing.

I managed to sign files using the SHA1 digest algorithm by using the following CryptoAPI functions (omitting parameters for brevity):

CertOpenStore
CertFindCertificateInStore
CryptAcquireCertificatePrivateKey
CryptCreateHash (with CALG_SHA1)
CryptHashData
CryptSignHash

The generated signature is compatible with "openssl dgst -sha1 -verify" (once the byte order is reversed).

My problem is: when I try to use CALG_SHA_256 with CryptCreateHash, it fails with error 80090008 (NTE_BAD_ALGID). By googling around, I found that I needed to use a specific provider (PROV_RSA_AES) instead of the default one. Since I would have a provider handle, I would also need to replace CryptAcquireCertificatePrivateKey by CryptGetUserKey. So I modified my program to look like:

CryptAcquireContext (with PROV_RSA_AES)
CertOpenStore
CertFindCertificateInStore
CryptGetUserKey
CryptCreateHash (with CALG_SHA256)
CryptHashData
CryptSignHash

Unfortunately, this didn't work as expected: CryptGetUserKey failed with error 8009000D (NTE_NO_KEY). If I remove the CryptGetUserKey call, the program runs until CryptSignHash, which fails with error 80090016 (NTE_BAD_KEYSET). I know the keyset does exist and works fine, since I was able to use it to sign the SHA1 digest.

I tried acquiring the context again with information from the certificate context I got from CertFindCertificateInStore: the best I could do was a successful CryptGetUserKey call, but CryptSignHash would always fail with the same error.

The private key I am trying to use is 2048 bits long, but I don't expect it to be a problem since it works with the SHA1 digest. I am at a loss, so any suggestion would be very welcome!

Answer 1

The problem is most likely to be that certificates on Windows "know" in which provider their private keys are stored. When you import your cert it would put the key into a certain provider type (probably PROV_RSA_FULL), when you then later try to access the key via the certificate it will probably end up in the same provider type.

You probably need to open the associated context for the certificate (have a look at CertGetCertificateContextProperty with the CERT_KEY_PROV_HANDLE_PROP_ID option). With that handle yyou could try exporting the key from the original provider context and reimporting into a new PROV_RSA_AES one (assuming the key is exportable).

Answer 2

80090008 is caused because of Base provider does not support SHA256, SHA384 and SHA512, you got to use CryptAcquireContext(hProv, NULL, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, 0);

Answer 3

Most of the previous answers have parts of the real answer. The way to do this is to get a HCRYPTPROV that uses the key container and the "Microsoft Enhanced RSA and AES Cryptographic Provider" by calling

CryptAcquireContext(&hCryptProv, <keyContainerName>, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, CRYPT_SILENT)

The resulting hCryptProv can be used to sign hashes created with all the supported SHA2: 256, 384, 512.

The Key Container Name can be obtained either with CertGetCertificateContextProperty() with argument CERT_KEY_PROV_INFO_PROP_ID if the key is obtained through the certificate, or by CryptGetProvParam() with argument PP_CONTAINER if there's already a HCRPYPTPROV for that key ( for example, obtained with CryptAcquireCertificatePrivateKey).

This technique works even if the private key is not exportable.

Answer 4

Once you have found the certificate, try calling CertGetCertificateContextProperty with CERT_KEY_PROV_INFO_PROP_ID to get the name of the provider and container with the key. Try calling CryptAcquireContext with these names, but specifying PROV_RSA_AES as the provider type.

You might also try replacing the provider name with "Microsoft Enhanced RSA and AES Cryptographic Provider", but this will definitely not work unless the provider is one of the other Microsoft providers.

Answer 5

I encountered a similar problem recently before I found this post. Although this post is helpful in comprehension of the problem, it didn't give out a solution. Finally, with the help of Google, I solved the problem. Although this question was asked 5 years ago, I write my solution here in case it will help any other guy.

First, let me describe my problem: We've ported OpenSSL 0.9.8 on our device acting as a SSL server, while we also provided a program running on Microsoft Windows as a SSL client, which also using OpenSSL library. This client/server model supporting client certificate authentication. The client call the Microsoft Crypto-API implementing RSA_method in OpenSSL to support the certificate authentication. So far, so good before we upgrading the OpenSSL library from 0.9.8 to 1.0.2 to support TLSv1.2. With TLSv1.2 chosen, the client certificate authentication always failed. After debugging, I found the problem is in the RSA_sign method of the client program. This method sign the hash result to get the digital signature which will be used in the SSL Client Verify message. The signing operation is calculated using CertFindCertificatePrivateKey/CryptCreateHash/CryptSetHashParam/CryptSignHash APIs as this question described. The error occurred in the CryptCreateHash call, where the requested hash method is SHA-384 but the hCryptoProv retrieved by CertFindCertificatePrivateKey corresponded to a CSP (Microsoft Enhanced Cryptographic Provider v1.0) does NOT support SHA-2 hash methods.

This is a real tedious description. If you are still continue reading, I guess you’ve met a similar problem too. Let me introduce you my solution.

My first reaction was to adjust the SSL negotiation param to downgrade the hash algorithm to SHA-1, but I failed. It seems SHA-2 hash is obligatory for TLSv1.2. Also did I try to achieve the sign operation by encrypting with the certificate private key, which was also failed. Crypto-API provides NO interface for certificate private key encryption (I’m not sure of this, but I did not found it.) After two days of resultless trying, I found this webpage: http://www.componentspace.com/Forums/Topic1578.aspx. It’s really like the light at the end of the tunnel. The certificate is binding with a CSP not supporting SHA-2, if we can convert it to a CSP supporting SHA-2, everything will be OK. In case of broken link, I paste the converting method here:

openssl pkcs12 -export -in idp.pem -out new-idp.pfx -CSP "Microsoft Enhanced RSA and AES Cryptographic Provider"

I just rebuilt the .pfx certificate file as the webpage said; re-imported the certificate. Then the problem is solved.

Hope this is helpful. :-)