C Libmcrypt cannot encrypt/decrypt successfully

Question

I am working with libmcrypt in c and attempting to implement a simple test of encryption and decryption using rijndael-256 as the algorithm of choice. I have mirrored this test implementation pretty closely to the man pages examples with rijndael as opposed to their chosen algorithms. When compiled with the string gcc -o encryption_test main.c -lmcrypt, the following source code produces output similar to: The encrypted message buffer contains j��A��8 �qj��%`��jh���=ZЁ�j The original string was ��m"�C��D�����Y�G�v6��s��zh�

Obviously, the decryption part is failing, but as it is just a single function call it leads me to believe the encryption scheme is not behaving correctly as well. I have several questions for the libmcrypt gurus out there if you could point me in the right direction.

First, what is causing this code to produce this broken output?

Second, when dealing with mandatory fixed-sizes such as the key size and block-size, for example a 256-bit key does the function expect 32-bytes of key + a trailing null byte, 31-bytes of key + a trailing null byte, or 32-bytes of key with the 33rd byte being irrelevant? The same question holds true for block-size as well.

Lastly, one of the examples I noted used mhash to generate a hash of the key-text to supply to the encryption call, this is of course preferable but it was commented out and linking in mhash seems to fail. What is the accepted way of handling this type of key-conversion when working with libmcrypt? I have chosen to leave any such complexities out as to prevent further complicating already broken code, but I would like to incorporate this into the final design. Below is the source code in question:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <mcrypt.h>

int main(int argc, char *argv[])
{
MCRYPT mfd;
char *key;
char *plaintext;
char *IV;
unsigned char *message, *buffered_message, *ptr;
int i, blocks, key_size = 32, block_size = 32;

message = "Test Message";

/** Buffer message for encryption */    
blocks              = (int) (strlen(message) / block_size) + 1;
buffered_message    = calloc(1, (blocks * block_size));

key = calloc(1, key_size);
strcpy(key, "&*GHLKPK7G1SD4CF%6HJ0(IV#X6f0(PK");

mfd = mcrypt_module_open(MCRYPT_RIJNDAEL_256, NULL, "cbc", NULL);

if(mfd == MCRYPT_FAILED)
{
    printf("Mcrypt module open failed.\n");
    return 1;
}

/** Generate random IV */
srand(time(0));
IV = malloc(mcrypt_enc_get_iv_size(mfd));
for(i = 0; i < mcrypt_enc_get_iv_size(mfd); i++)
{
    IV[i] = rand();
}

/** Initialize cipher with key and IV */
i = mcrypt_generic_init(mfd, key, key_size, IV);
if(i < 0)
{
    mcrypt_perror(i);
    return 1;
}

strncpy(buffered_message, message, strlen(message));    
mcrypt_generic(mfd, buffered_message, block_size);
printf("The encrypted message buffer contains %s\n", buffered_message);
mdecrypt_generic(mfd, buffered_message, block_size);
printf("The original string was %s\n", buffered_message);
mcrypt_generic_deinit(mfd);
mcrypt_module_close(mfd);
return 0;
}

Answer 1

You need to re-initialize the descriptor mfd for decryption, you cannot use the same descriptor for both encryption and decryption.