You aren't clear which files you combined, but it should work to use openssl to combine the cert and private key to a PKCS#12:
cat cert_public_key.pem cert_private_key.pem >combined.pem
openssl pkcs12 -export -in combined.pem -out cert.p12
or on the fly but (update:) the privatekey must be first:
cat cert_private_key.pem cert_public_key.pem | openssl pkcs12 -export -out cert.p12
If your cert needs any chain cert(s) -- the CA should have told you this when you submitted the CSR and they issued the cert -- it's easiest to also include it(them) now.
Then (1) some Java programs can actually use a pkcs12 directly as a keystore, but (2) if you need or prefer a JKS use keytool:
keytool -importkeystore -srckeystore cert.p12 -srcstoretype pkcs12 -destkeystore cert.jks
If you care about the alias in the resulting JKS, easiest to fix it after converting.
Also: just changing the labels in an encrypted PEM doesn't unencrypt it, nor does changing the label from generic PKCS#8 to RSA actually change the data to match (and they are different, though only a little). If you do want a separate PEM file with the decrypted private key:
openssl pkey -in encryptedpk8 -out clearpk8.pem # 1.0.0 up
openssl pkcs8 -in encryptedpk8 -out clearpk8.pem # 1.0.0 up
openssl pkcs8 -topk8 -nocrypt -in encryptedpk8 -out clearpk8.pem # below 1.0.0
openssl rsa -in encryptedpk8 -out clearrsa.pem