Signature verification with OpenSSL

Question

I'm attempting to use OpenSSL to verify a signature that was created using SHA256 with RSA (specifically the Java implementation in Signature.getInstance("SHA256withRSA")). I also have strings representing the hexadecimal forms of the public key exponent and modulus. (For example, the exponent is "010001", corresponding to 65537.) Here's the code.

int verify(string &plaintext, string &exp, string &mod, string &sig) {
  RSA *pub_key = RSA_new();
  if (!BN_hex2bn(&pub_key->n, mod.c_str()))
    exit(1);
  if (!BN_hex2bn(&pub_key->e, exp.c_str()))
    exit(1);
  int verified = RSA_verify(
      NID_sha256,
      reinterpret_cast<const unsigned char *>(plaintext.data()),
      plaintext.size(),
      reinterpret_cast<const unsigned char *>(sig.data()),
      sig.size(),
      pub_key);
  RSA_free(pub_key);
  return verified;
}

I also tried SHA256-hashing the plaintext first and passing that to RSA_verify (along with SHA256_DIGEST_LENGTH), but that didn't work either.

unsigned char hash[SHA256_DIGEST_LENGTH];
SHA256_CTX sha256;
SHA256_Init(&sha256);
SHA256_Update(&sha256, plaintext.data(), plaintext.size());
SHA256_Final(hash, &sha256);
// use hash instead of plaintext in call to verify

Unfortunately, both implementations return 0. Am I doing anything wrong here, or does it seem likely that one of my inputs is wrong?

Answer 1

Okay, I figured it out. It turns out my input was bad. The signature wasn't encoded the way I thought it was.

Once I got the right signature, I had to use hash and SHA256_DIGEST_LENGTH as the message inputs.