This issue was caused because an exception occurred when API CryptMsgControl was called.
Why it works on win7 but doesn't on win8?
This is because the imported cert by the code is using KeySpec AT_SIGNATURE. This is not the proper spec for encryption/decryption. It is meant for signing. AT_KEYEXCHANGE is the proper one for this purpose.
In Win7 the improper spec works due to the specific implementation for parsing the legacy CAPI keys not strictly enforcing key usages from the KeySpec. Hence, AT_SIGNATURE allowed encrypt/decrypt usages.
To solve this problem, we need to import the cert using certutil command tool with following command:
certutil -user –p -importpfx MY AT_KEYEXCHANGE
We just need to make sure the pfx file has KeySpec set to 1 (AT_KEYEXCHANGE). To check this, we can use the command certutil.exe -dump -v xyz.pfx
When we import a pfx using certutil with KeySpec AT_KEYEXCHANGE specified, we are modifying the KeySpec property of the pfx file. After that, we can export it to a pfx file again, then we can check the KeySpec again with the command mentioned above, and we will see the KeySpec change to AT_KEYEXCHANGE.
Some more details you can find here