Loading a RSA private key from memory using libxmlsec

Question

I'm currently using libxmlsec into my C++ software and I try to load a RSA private key from memory. To do this, I searched trough the API and found this function.

It takes binary data, a size, a format string and several PEM-callback related parameters.

When I call the function, it just stucks, uses 100% of the CPU time and never returns. Quite annoying, because I have no way of finding out what is wrong.

Here is my code:

d_xmlsec_dsig_context->signKey =
    xmlSecCryptoAppKeyLoadMemory(
        reinterpret_cast<const xmlSecByte*>(data),
        static_cast<xmlSecSize>(datalen), 
        xmlSecKeyDataFormatBinary,
        NULL,
        NULL,
        NULL
    );

data is a const char* pointing to the raw bytes of my RSA key (using i2d_RSAPrivateKey(), from OpenSSL) and datalen the size of data.

My test private key doesn't have a passphrase so I decided not to use the callbacks for the time being.

Has someone already done something similar ? Do you guys see anything that I could change/test to get forward on this problem ?

I just discovered the library on yesterday, so I might miss something obvious here; I just can't see it.

Thank you very much for your help.

Answer 1

I changed the format of data to PEM, using the OpenSSL function PEM_write_bio_RSAPrivateKey() and changed the third argument to the call to xmlSecCryptoAppKeyLoadMemory() so it matches the new format.

The new code is:

d_xmlsec_dsig_context->signKey =
xmlSecCryptoAppKeyLoadMemory(
    reinterpret_cast<const xmlSecByte*>(data), // data is now in PEM format
    static_cast<xmlSecSize>(datalen), 
    xmlSecKeyDataFormatPem, // Updated
    NULL,
    NULL,
    NULL
);

And since then, everything works: the call does no longer get stuck.