Keep in mind that encodings are for human-computer interactions, ciphers are for computer-computer interaction. When building a cipher, you actually create a bit stream, which has no inherent encoding.
To compensate Ruby's tendency of interpreting strings with encoding, you could transform the values to Base64, like so:
require 'base64'
module MyApp::XVP
def xvp_password_encrypt_vnc64(hex)
Base64.strict_encode64 xvp_password_encrypt_vnc(hex)
end
def xvp_password_decrypt_vnc64(hex)
xvp_password_decrypt_vnc Base64.strict_decode64(hex)
end
end
and perform your tests on the output of these methods.
Another possibility would be to convert your spec data to Encoding::BINARY
(which is an alias for Encoding::ASCII_8BIT
):
context 'decoding password'
let(:encoded) { "\x88\x90r\"\x9EN\xFFR".force_encoding('BINARY') }
let(:decoded) { "L1UkDr]c" }
subject { MyApp::XVP::xvp_password_decrypt_vnc(encoded) }
it { should eq decoded }
end