OpenSSL dgst authentication failure

Question

I've tried to replicate the workflow presented on this blog in OpenSSL: http://farid.hajji.name/blog/2009/07/27/public-key-cryptography-with-openssl/

However, authentication seems to fail despite many variations. What's going wrong? Please see the code below you can copy and paste into OpenSSL. Note that I am using the compiled Windows binary version of OpenSSL.

//================Phase 1 - Setup================

//Generate my private key (myprivatekey.txt)
genpkey -algorithm RSA -out C:\myprivatekey.txt -pass pass:abc123 -pkeyopt rsa_keygen_bits:2048

//Generate friend's private key (friendprivatekey.txt)
genpkey -algorithm RSA -out C:\friendprivatekey.txt -pass pass:123abc -pkeyopt rsa_keygen_bits:2048

----------------

//Extract my public key (mypublickey.txt) from my private key (myprivatekey.txt)
rsa -passin pass:abc123 -in C:\myprivatekey.txt -pubout -out C:\mypublickey.txt

//Extract friend's public key (friendpublickey.txt) from my private key (friendprivatekey.txt)
rsa -passin pass:123abc -in C:\friendprivatekey.txt -pubout -out C:\friendpublickey.txt

----------------

//Generate my password (a random base64 string password saved mypassword.txt)
rand -base64 -out C:\mypassword.txt 128

//Generate friend's password (a random base64 string password saved to friendpassword.txt)
rand -base64 -out C:\friendpassword.txt 128

//Delete the .rnd file that's generated?  Not sure what it is.

----------------

//Encrypt my password using my private key (encrypted password saved to a binary file - myencryptedpassword.txt)
pkeyutl -in C:\mypassword.txt -out C:\myencryptedpassword.txt -inkey C:\myprivatekey.txt -passin pass:abc123

//Encrypt friend's password using friend's private key (encrypted password saved to a binary file - friendencryptedpassword.txt)
pkeyutl -in C:\friendpassword.txt -out C:\friendencryptedpassword.txt -inkey C:\friendprivatekey.txt -passin pass:123abc

----------------

//Convert my encrypted password to base64 from binary (saved as myencryptedpasswordbase64.txt)
base64 -in C:\myencryptedpassword.txt -out C:\myencryptedpasswordbase64.txt

//Convert friend's encrypted password to base64 from binary (saved as friendencryptedpasswordbase64.txt)
base64 -in C:\friendencryptedpassword.txt -out C:\friendencryptedpasswordbase64.txt

----------------

//Create a signed hash of my password so my friend knows it's coming from me (signed hash saved as mysignedhash.txt and is in binary form)
dgst -sha256 -sign C:\myprivatekey.txt -passin pass:abc123 -out C:\mysignedhash.txt C:\myencryptedpasswordbase64.txt

//Create a signed hash of friend's password so I know it's coming from my friend (signed hash saved as friendsignedhash.txt and is in binary form)
dgst -sha256 -sign C:\friendprivatekey.txt -passin pass:123abc -out C:\friendsignedhash.txt C:\friendencryptedpasswordbase64.txt

----------------

//Convert my signed hash from binary to base64
base64 -in C:\mysignedhash.txt -out C:\mysignedhashbase64.txt

//Convert friend's signed hash from binary to base64
base64 -in C:\friendsignedhash.txt -out C:\friendsignedhashbase64.txt

//================Phase 2 - Authentication================

//Now, we reverse the process and authenticate the friend.  Let's prefix all output files with "phase2"

//I provide friend with my public key and my encrypted password 
//Friend provides me with their public key

//Convert friend's encrypted password from base64 to binary.  The output file will be the same as friendsignedhash.txt
base64 -d -in C:\friendsignedhashbase64.txt -out C:\phase2friendsignedhash.txt

//Convert friend's signed hash from base64 to binary.  The output file will be the same as C:\friendsignedhash.txt
base64 -d -in C:\friendencryptedpasswordbase64.txt -out C:\phase2friendencryptedpassword.txt

//Verify if the password originates from my friend (by checking against my friend's public key)
dgst -sha256 -verify C:\friendpublickey.txt -signature C:\phase2friendsignedhash.txt -out C:\friendresult.txt C:\phase2friendencryptedpassword.txt

Any idea of why the verification failure occurs?

Answer 1

The problem is that you signed friendencryptedpasswordbase64.txt, but are trying to verify using phase2friendencryptedpassword.txt (non-base64). If you verify with the former, it works perfectly!

In addition to that I do see a couple security-related problems. They aren't causing the issue you asked about, but they go against good security practice.

First, you probably want to encrypt your password with your friend's public key instead of your private key. Then only your friend's private key will be able to decrypt it. Otherwise anyone with your public key can decrypt your encrypted password.

Second, generally, you don't want to use the same key for multiple purposes (i.e. signing and encrypting). Use a separate key for each function.