The first thing I spot is your second rotation in the decryption function:
rol al,3
This should probably be:
ror al, 3
The second thing is the xor
:
xor eax,edx
which should probably be moved to before the second rotation and mov
, and also change the register that is xored:
xor ecx,edx
mov eax,ecx
ror al,3
To summarize the decryption:
push ebp
mov ebp,esp
push edx
push ecx
push eax
movzx eax,byte ptr [eax]
mov edx,eax
pop eax
mov byte ptr [eax],dl
pop ecx
rol al,3
xor ecx,edx
mov eax,ecx
ror al,3
pop edx
pop ebp
ret
Edit
Your encryption order is: rol, xor, ror
This yields the decryption order: rol, xor, ror
In fact you should be able to run the encryption routine twice and it would yield the original input (plaintext).