Why would you want to build your own solution to this kind of hybrid cryptography? There are already existing standards and methods that can help you.
I recommend you look into the PKCS#7 standard, which is the basis for S/MIME. OpenSSL has a direct interface to it. You tell it which data you want to encrypt using an asymmetric key and it handles the rest for you.
Look at pkcs7_encrypt
and pkcs7_decrypt
as well as the i2d_PKCS7_*
functions for how to extract that data into a transmittable format (and d2i_PKCS7_*
for reverse). See OpenSSL's documentation: PKCS7_encrypt, PKCS7_decrypt and you might want to familiarize yourself with the i2d/d2i convention used by OpenSSL (this is X509 but the d2i part applies here as well.
Edit: Here is an example for encryption (decryption is analogous):
#include <stdio.h>
#include <openssl/pem.h>
#include <openssl/crypto.h>
#include <openssl/err.h>
int main()
{
STACK_OF(X509) *certs;
FILE *fp;
BIO *bio;
PKCS7 *p7;
ERR_load_crypto_strings();
OpenSSL_add_all_algorithms();
certs = sk_X509_new_null();
fp = fopen("cert.pem", "r");
sk_X509_push(certs, PEM_read_X509(fp, NULL, NULL, NULL));
fclose(fp);
bio = BIO_new_file("data.txt", "r");
p7 = PKCS7_encrypt(certs, bio, EVP_des_ede3_cbc(), 0);
BIO_free(bio);
bio = BIO_new_file("data.txt.enc", "w");
i2d_PKCS7_bio(bio, p7);
BIO_flush(bio);
BIO_free(bio);
ERR_print_errors_fp(stdout);
return 0;
}
I have uploaded the full example to my repository.