I am an idiot. At the start of this project I knew that the signature format had to be either DER or PEM. I thought I had configured this correctly, but somehow it ended up in the situation where the Jarsigner's signature was in DER format when I wanted to verify a PEM signature.
My solution is to always expect a DER signature. This is default for the Jarsigner. For my OpenSSL signer/verifier I had to make sure the outform and inform was der: -outform der and -inform der.
Code wise I had to change this:
if (!(p7 = PEM_read_bio_PKCS7(in, NULL, NULL, NULL))) {
into this:
if (!(p7 = d2i_PKCS7_bio(in, NULL))) {