The source of the PK11_GenerateRandom()
function: http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/pk11wrap/pk11slot.c#2285
Based on my calculation the -8023 corresponds error SEC_ERROR_PKCS11_DEVICE_ERROR
The reason (thanks to jariq's hints) is described here: https://bugzilla.mozilla.org/show_bug.cgi?id=331096
It is that in the past, it was okay to fork and continue using PKCS11 functions. They decided that it cannot be like that, and now, conclusion is that the parent should not initialize these functions if a child after forking is expected to use them.
PKCS11 internal functions are checking if there was forking (they use various methods dependently on what platform the code is built.) For example, they stored pid of the process in an internal memory, and in some expensive functions or called not so often they compare this preserved pid with current getpid()
.
The fix for our problem will require redesigning the code.