What is this undocumented flag in the Microsoft CryptoAPI File Encrypt Sample?

Question

In the Microsoft C sample code for RC4-encrypting a file using the CryptoAPI, the functions CryptGenKey and CryptDeriveKey are passed an undocumented flag

#define KEYLENGTH     0x00800000
...
if (CryptGenKey(
        hCryptProv, 
        ENCRYPT_ALGORITHM, 
        KEYLENGTH | CRYPT_EXPORTABLE, 
        &hKey))
...    

There is no flag with this value defined in the CryptoAPI header file wincrypt.h. Leaving it out doesn't seem to do any harm. In fact, when I change the algorithm from RC4 to AES, using this flag results in an ERROR_INVALID_PARAMETER.

What is it there for?

Answer 1

It's not undocumented, and it's not a flag. From the page you linked for CryptGenKey (emphasis mine):

Specifies the type of key generated. The sizes of a session key, RSA signature key, and RSA key exchange keys can be set when the key is generated. The key size, representing the length of the key modulus in bits, is set with the upper 16 bits of this parameter. Thus, if a 2,048-bit RSA signature key is to be generated, the value 0x08000000 is combined with any other dwFlags predefined value with a bitwise-OR operation. The upper 16 bits of 0x08000000 is 0x0800, or decimal 2,048. The RSA1024BIT_KEY value can be used to specify a 1024-bit RSA key

The #define provides the key size used to represent the length of the key modulus in bytes in the upper 16 bits, exactly like the above states. The code sample you've included uses 0x00800000 instead, to indicate a 128-bit key.

The quote goes on to explain:

The upper 16 bits of 0x08000000 is 0x0800, or decimal 2,048.

The key size (in the upper 16 bits) is combined with a bitwise OR of one of the predefined flag values.