Why would PK11_GenerateRandom() return an error -8023?

StackOverflow https://stackoverflow.com/questions/22303198

  •  12-06-2023
  •  | 
  •  

Question

I am looking through the Internet trying to find the source of the PK11_GenerateRandom() function to see why would the function fail. I have a program that perfectly uses this function but when we moved to a new flavor of Linux, it fails after forking (fork()) Since I do not believe there is a problem with NSS, I suspect that we are doing something incorrectly which was disregarded in the older versions of Linux but with the new one there is an issue.

The OpenSSL package is the same on the 'good' and the 'bad' server:

OpenSSL 0.9.8e-fips-rhel5 01 Jul

NSS rpm differs though. The 'good' has 

nss-3.12.2.0-2.el5

and the bas has this version

nss-3.15.3-4.el5_10

The 'good' server uses quite obsolete Linux:

Linux GOOD 2.6.18-128.el5 #1 SMP Wed Jan 21 08:45:05 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
Enterprise Linux Enterprise Linux Server release 5.3 (Carthage)
Red Hat Enterprise Linux Server release 5.3 (Tikanga)

The 'bad' server is newer:

Linux BAD 2.6.18-371.4.1.el5 #1 SMP Wed Jan 29 11:05:49 PST 2014 x86_64 x86_64 x86_64 GNU/Linux
Oracle Linux Server release 5.10
Red Hat Enterprise Linux Server release 5.10 (Tikanga)

Any clue as to where could I find the source or what could be the reason for failure (like side effect coming from fork()) would be greatly appreciated.

  • Greg

Edit

Here is the code, which is so simple, that I did not think it is needed.

/* random points to properly allocated memory, let=32 */
SECStatus rv = PK11_GenerateRandom((unsigned char *)random, (int)len);
if ( rv != SECSuccess )
    printf( "PK11_GenerateRandom error = %d\n", PR_GetError()) ;

and the output message is, of course:

PK11_GenerateRandom error = -8023
  • Greg
Was it helpful?

Solution

The source of the PK11_GenerateRandom() function: http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/pk11wrap/pk11slot.c#2285

Based on my calculation the -8023 corresponds error SEC_ERROR_PKCS11_DEVICE_ERROR

The reason (thanks to jariq's hints) is described here: https://bugzilla.mozilla.org/show_bug.cgi?id=331096

It is that in the past, it was okay to fork and continue using PKCS11 functions. They decided that it cannot be like that, and now, conclusion is that the parent should not initialize these functions if a child after forking is expected to use them.

PKCS11 internal functions are checking if there was forking (they use various methods dependently on what platform the code is built.) For example, they stored pid of the process in an internal memory, and in some expensive functions or called not so often they compare this preserved pid with current getpid().

The fix for our problem will require redesigning the code.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top