Months later I happened to figure out the answer to my own question. For anyone else with the same issue, here is what I did:
Convert your existing private key and the CA signed cert into a pkcs12 format (this is required since Java's keytool doesn't allow the direct importation of these items). This can be accomplished in a single openssl command:
openssl pkcs12 -export -name signing -in signing.cert -inkey myPrivateKey.key -out keystore.p12
Where signing is the name of my pkcs12 keystore, signing.cert is my CA supplied signed cert, and (obviously) myPrivateKey.key is my private key that was used to sign the Cert Request.
Import this newly created keystore into a Java keystore:
keytool -importkeystore -destkeystore keystore -srckeystore keystore.p12 -srcstoretype pkcs12 -alias signing
Import your CA's Java cert into the keystore. I'm not exactly sure what magic this does but without it the cert chain isn't followed (even when manually adding intermediate certs). This cert is usually provided via the email where your signing cert arrived in. For our purposes it's called signing.pkcs7.
keytool -importcert -file signing.pkcs7 -keystore keystore -v -alias signing
You'll have to enter the keystore password you used when creating the Java keystore.
Use the maven-jarsigner-plugin (or whatever automation is required) to sign your projects during build.