Found it, stop looking!
The problem arises from the fact that the PKCS12 format does not support separate passwords for the storage file and any keys inside it. Both the storage file and the key inside it have to be encrypted with the same password.
So the easiest was to go through the process again, and in the step of generating the initial private key and keystore with keytool:
keytool -genkey -alias myapp -validity 1095 -keyalg RSA -keystore keystore.jks
make sure to use the same password for both the keystore (storepass) and the private key (keypass).
As an alternative, if you really must/want to use separate passwords for the keystore and key in that step, in the initial step for creating the PEM:
keytool -importkeystore -srckeystore [MY_KEYSTORE.jks]
-destkeystore [MY_FILE.p12]
-deststoretype PKCS12
-deststorepass [PASSWORD_PKCS12] change this to [KEYPASS]
-srcstorepass [PASSWORD_JKS]
set the -srcstorepass to the password of the keystore (storepass), but set the -deststorepass of the PKCS12 file to the password of the private key (keypass) used when creating the key/keystore via the keytool -genkey command further above.
I have not tested this alternative, but it should work as well, since both the PCKS12 store and the key would end up using the same password.