This is how you can add a private key to your new keystore:
...
char[] password = "123".toCharArray();
ks.load(null, password);
PrivateKeyEntry entry = new PrivateKeyEntry(pair.getPrivate(),
new java.security.cert.Certificate[]{PKCertificate});
ks.setEntry("hive",entry , new KeyStore.PasswordProtection("123".toCharArray()));
fos = new FileOutputStream("hive-keystore.pkcs12");
ks.store(fos, password);
fos.close();
I've tried it with jarsigner and it gave no errors, so it should be fine.