OpenSSL Encryption of Session Key

Question

I am writing a method that encrypts session keys. It needs to do this such that the key can be decrypted by a different program that has been tested successfully. The decryption program cannot change. Where I am stuck is on getting my encryption to work in a way that it aligns with the decryption routine.

Let me give the decryption routine first. Remember, this cannot change:

public Boolean decryptSessionKey() {

    // first, base64 decode the session key
    String sslString = "openssl base64 -d -in enc_sesskey -out temp";

    try {
        Process p = Runtime.getRuntime().exec(sslString);
    } catch (IOException e2) {
        e2.printStackTrace();
    }

    // now we can decrypt it
    try {
        sslString = "openssl rsautl -in temp -inkey privkey.pem -decrypt";
        Process p = Runtime.getRuntime().exec(sslString);   
        BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));

        try {
            String s;
            while ((s = stdInput.readLine()) != null) {           
                decrypted_session_password = s;
                writeToFile(decrypted_sesskey, s);
            }
            return true;

        } catch (Exception e) {
            return false;
        }
    } catch (IOException e1) {
        return false;
    } catch (Exception e) {
        return false;
    }
}

Here is the encryption routine that I am writing. I yields base64 encoded text, but it is ultimately not decryptable. It's worth noting that I have verified that the decryption routine correctly picks up the results of the encryption routine (no handshake problems between the two).

public Boolean encryptSessionKey(Cert receiver_cert) {

    String sslString = 
        "openssl rsautl base64 -in sesskey -out temp -inkey cert.pem -encrypt -certin";

    // run this openssl encryption. Note that it will not yet be base64 encoded
    try {
        Process p = Runtime.getRuntime().exec(sslString);
    } catch (IOException e2) {
        e2.printStackTrace();
    }

    // now we base64-encode the encrypted file
    sslString = "openssl base64 -in temp -out enc_sesskey"; 

    try {
        Process p = Runtime.getRuntime().exec(sslString);   
    } catch (IOException e1) {return false;
    } catch (Exception e) {return false;
    }

    return true;
}

I'm really stuck. Any help is appreciated. Thank you.

Answer 1

RSA encryption with PKCS padding and public key acquired from certificate:

openssl rsautl -encrypt -in sesskey -inkey cert.pem -certin -out temp
openssl base64 -e -in temp -out enc_sesskey

RSA decryption with PKCS padding and private key:

openssl base64 -d -in enc_sesskey -out temp
openssl rsautl -decrypt -in temp -inkey privkey.pem -out sesskey2

Tested and confirmed to be OK because content of the "sesskey" file is the same as content of the "sesskey2" file.