How to convert OpenSSL SHA-512 hash to GMP number to use in RSA encryption method

Question

SHA1(data, length, hash);
unsigned char *labelPtr;

labelPtr = hash;
mpz_set_str (encrypted, labelPtr, 16);
gmp_printf("hashed= %Zd\n", encrypted);

encrypted equals 0? I need to get an integer from this hash so that I can perform RSA encryption. I see the hash when I debug, but cannot seem to turn this into an int. It is SHA-512 so 512 bits? Weird characters come out of printf("%s",hash); atoi gives 0 too, do I have to break up the hash by character? That would work I think.

Answer 1

mpz_set_str second parameter must be a string.

The hash argument you pass to SHA1 function is probably not a string. Convert it to a string format before passing it to the mpz_set_str function.