pkcs#11 memory error - what might be the common reasons?

Question

I am getting the CKR_DEVICE_MEMORY error code for C_Encrypt call using cryptoki library.

From the PKCS#11 spec, CKR_DEVICE_MEMORY means The token doesnot have sufficient memory to perform the requested function.

Under what circumstances, do we get the token's memory completely full?

The HSM has been working 24x7 for 7 days continuously mostly encrypting and decrypting files during the day time with 2 parallel sessions. I haven't called C_Finalize in the last 7 days. so cryptoki library has been working in its memory space from the point it has been initialised(see a related post on this).

I can see from my applications, debug log, what ever, i am allocating, i am deallocating so there is no memory leak from my application code.

UPDATE 1: There is a related detailed discussion on how i can call C_Finalize in Application_Endof the ASP.NET. The main reason i couldn't use this because after recycling/timeout, the ASP.net threads access a single session resulting in CKR_OPERATION_ACTIVE error. In my case multiple applications are accessing the HSM via a webservice.

Answer 1

You mention here that you are not closing your sessions. If that is true, that is most probably the cause of the CKR_DEVICE_MEMORYs.

Answer 2

Let’s consider the memory of HSM and host computer (which Cryptoki library runs on) separately. Correctly implemented Cryptoki library should return CKR_DEVICE_MEMORY if HSM device suffers from the lack of memory and CKR_HOST_MEMORY if Cryptoki library can’t allocate host computer memory for its internal structures (if it is implemented as a shared library then the process can’t allocate memory). So if your pkcs11 library is implemented correctly then CKR_DEVICE_MEMORY means insufficient device (HSM) memory literally. There are a lot of reasons of such bugs. We can’t consider all branches. It’s possible to restrict some issues only. Answering your question there are three main common reasons of problems with memory in Cryptoki library:

1. Memory for crypto operations. Client of Cryptoki is responsible for allocating such memory, not Cryptoki library. For example client of Cryptoki library must allocate buffer for final result before invoking C_EncryptFinal. If buffer size is not enough then Cryptoki returns CKR_BUFFER_TOO_SMALL.
2. HSM memory. CKR_DEVICE_MEMORY points to this case but it's beyond control of most software developers.
3. Memory for internal service structures in Cryptoki library. For example when you open session the memory for this structure is allocated. When you stop encryption process and start decryption within the same session the mode for this session changes. Cryptoki library should support internal state between calls because it supports multi-part iterative operations. When switching from one kind of operation to another it should free previous structures and allocate new ones in memory like heap. If application developer has library sources or wants to help in finding error it worth to do following in this situation(for this particular incident assuming library erroneously reports CKR_DEVICE_MEMORY instead of CKR_HOST_MEMORY). Try to run program only for one kind of operation (say encryption). If it works without memory error for mentioned period of time then it’s possible that memory leaks occurres while changing operation types. But you says that:"one session for encryption and the other for decryption". It narrows the scope. Probably memory for storing the state for multi-part operation leaks. Monitor the amount of memory after several operations. If you don't use multi-part operations then most likely it's the case 2 because Cryptoki library in such circumstances shouldn't allocate any non-stack memory.

These estimates are only to illustrate general issues with memory in such libraries.

Answer 3

I have also this problem and year is 2020 :S .Net Framework + Rest Api couple have this problem this time. I'm using HSM for decrypt method. I have a login method interactive channel, and we need to make performance test. The service has an instance from Pkcs11

pkcs11 = new Pkcs11(hsmPath, true);
slot = GetUsableSlot(pkcs11);
TokenInfo tokenInfo = slot.GetTokenInfo();
session = slot.OpenSession(true);
session.Login(CKU.CKU_USER, userLoginPin);
secretKey = GenerateKey(session);

And this is the Decrypt method.

public byte[] Decrypt(byte[] encryptedTextByteArray) {

    Mechanism mechanism = new Mechanism(CKM.CKM_AES_ECB);
    byte[] sourceData = encryptedTextByteArray;
    byte[] decryptedData = null;

    using (MemoryStream inputStream = new MemoryStream(sourceData), outputStream = new MemoryStream())
    {
        try
        {                
            session.Decrypt(mechanism, secretKey, inputStream, outputStream, 4096);
        }
        catch (Pkcs11Exception ex)
        {
            throw;
        }
        decryptedData = outputStream.ToArray();
    }
    return decryptedData;
}

When I try to make performance test using Postman runner, there is no problem for one thread. If I increase thread count, It appears these errors. First error: CKR_OPERATION_ACTIVE Next error: CKR_DEVICE_MEMORY

I tried these methods. -For every request closed session. And also opened session for new request. But not succeeed. The same errors appeared. (Of course request and response time increased) -For evey request closed the conenction. And also opened new connection for new request. The same errors appeared. (Of course request and response time increased)

Anyone helps me? :)