Manual page states that: RSA_public_encrypt() returns the size of the encrypted data. RSA_private_decrypt() returns the size of the recovered plaintext. On error, -1 is returned; the error codes can be obtained by ERR_get_error(3).
Your code should look like this:
unsigned char plain [13] = "Hello World!";
unsigned char encrypted[1024]={};
unsigned char decrypted[1024]={};
int padding = RSA_PKCS1_OAEP_PADDING;
int flen = 13;
int res = RSA_public_encrypt(flen, plain, encrypted, rsa_pbk, padding);
flen = res;
res = RSA_private_decrypt(flen, encrypted, decrypted, this->rsa_pvk, padding);