How to use OpenSSL to validate a *.SF / *.RSA signature created by the Jarsigner

Question

I have an archive I want to sign and I want to validate it in C with OpenSSL.

To sign the archive the jarsigner seemed like a good idea, considering I wouldn't have to create something on my own, and it seems to work great. With OpenSSL I can validate the different digest values, but I can't get it to validate the *.SF *.RSA signature.

The steps I have taken:

Create a keystore

$ keytool -genkeypair -alias <alias> -keystore <keystore> -validity 360 -keyalg RSA -keysize 2048 -sigalg SHA256withRSA

Sign the archive

$ jarsigner -keystore <keystore> -signedjar <signedFile>.zip <fileToSign>.zip <alias>

Snipped of C validation code

BIO *in = NULL, *indata = NULL;
PKCS7 *p7 = NULL;
int flags = PKCS7_DETACHED;
    flags |= PKCS7_NOVERIFY;
    flags |= PKCS7_BINARY;

OpenSSL_add_all_algorithms();

/* load *.RSA (PKCS7) file */
if (!(in = BIO_new_file(path, "r"))) {
    printf ("Can't open input file %s\n", path);
    status = FAILURE;
}

if (!(p7 = PEM_read_bio_PKCS7(in, NULL, NULL, NULL))) {
    printf ("Error in reading PKCS7 PEM file.\n");
    status = FAILURE;
}

/* load *.SF file */
if (!(indata = BIO_new_file(path, "r"))) {
    printf("Can't read content file %s\n", path);
    status = FAILURE;
}

/* validate signature */
if (PKCS7_verify(p7, NULL, NULL, indata, NULL, flags))
    printf("Signature verification successful!\n");
else {
    printf("Signature verification failed!\n");
    status = FAILURE;
}

The error

It fails in "PEM_read_bio_PKCS7(...)".

I'm looking for either a way to validate it in the terminal or with C using OpenSSL. C is preferred ;) but I can always convert the command to code in case you only know how to do it manually.

Answer 1

I am an idiot. At the start of this project I knew that the signature format had to be either DER or PEM. I thought I had configured this correctly, but somehow it ended up in the situation where the Jarsigner's signature was in DER format when I wanted to verify a PEM signature.

My solution is to always expect a DER signature. This is default for the Jarsigner. For my OpenSSL signer/verifier I had to make sure the outform and inform was der: -outform der and -inform der.

Code wise I had to change this:

if (!(p7 = PEM_read_bio_PKCS7(in, NULL, NULL, NULL))) {

into this:

if (!(p7 = d2i_PKCS7_bio(in, NULL))) {

Answer 2

If you want to do check the certificate chain using command-line tools, here is how:

unzip -p your.jar META-INF/*.RSA | openssl pkcs7 -inform DER -text -print_certs