Understanding NSS PK11_CipherOp and C memory allocation

Question

Having some issues with C. I have this is my code:

// First line works, second line does not! 
char outbuf[1024];
// char *outbuf = (char *) malloc(1024);  // char is always 1

I am passing this outbuf to a method called PK11_CipherOp(), declared in the NSS library. The Documentation for this method can be found here, you can see that it expects a char * for the outbuf param.

I cannot understand why the first line works reliably, and the second line does not!

To see the full code, please see File Source

Thanks,

Hamilton

Answer 1

Your problem appears to be a missing declaration for malloc - you haven't included <stdlib.h>.

This means that your compiler is assuming the function returns int, which is then being explicitly cast to (unsigned char *). If you are on a platform where sizeof(int) != sizeof(void *), like x86-64, then this will mangle the pointer value.

This illustrates perfectly why in C (unlike C++) you should not cast the result of malloc() - doing so is unnecessary, and hides this exact bug from you. It also illustrates why you should turn on compiler warnings.

The fix is to #include <stdlib.h>, and remove the cast from malloc() while you're there.

Addendum:

Your other issue is that you're passing sizeof(outbuf) to PK11_CipherOp(). This will be correct (equal to out_buf_size) when outbuf is an array, but is incorrect (the size of the pointer itself) when outbuf is a pointer.

Replace each sizeof(outbuf) with out_buf_size and you should be good to go (there's another one later too)...

Answer 2

Found the issue. Turns out that I was using sizeof(outbuf) as one of the parameters. When the outbuf is declared on the stack, as char outbuf[1024]; this returns 1024. However, when outbuf is on the heap, as char * outbuf = malloc(1024); the size of outbuf is just 4. This parameter told the function how much room was available in the outbuf, so the function thought that it only had 4 bytes of room. It needs 16 to do anything so it just returned without performing any work.