To answer your specific case:
RFC3447 says that RSAES-PKCS1-v1_5 encryption scheme can operate on messages of length up to k - 11 octets (k is the octet length of the RSA modulus) so there is no way you can use 128bit RSA key to encrypt 192bit DES3 key. OpenSSL would end up with the following error:
openssl rsautl -encrypt -in symkey.data -out symkey.enc -pubin -inkey rsaPub.der -keyform DER -pkcs
RSA operation error
7240:error:0406D06D:rsa routines:RSA_padding_add_PKCS1_type_2:data too large for key size:.\crypto\rsa\rsa_pk1.c:151:
Are you sure that you have provided correct information in your question?
To answer key wrapping with OpenSSL in general:
Yes it is possible to wrap/unwrap DES3 key with the RSA key using OpenSSL. Mechanism CKM_RSA_PKCS you are using with your PKCS#11 library identifies RSAES-PKCS1-v1_5 encryption scheme (RSA encryption with PKCS#1 v1.5 padding) which is fully supported by OpenSSL.
You can easily check out that my answer is correct with the command line OpenSSL tool and a few lines of code that use PKCS#11 library:
Generate RSA key pair:
openssl genrsa -out private.key 2048
Extract RSA public key from generated RSA key pair:
openssl rsa -in private.key -pubout -out public.key
Extract modulus and exponent from RSA public key:
openssl rsa -in public.key -pubin -text
Import RSA public key into your PKCS#11 token, generate DES3 key, wrap DES3 key with RSA public key and save results (code is written in C#):
using (Pkcs11 pkcs11 = new Pkcs11("siecap11.dll", false)) { // Find first slot with token present Slot slot = pkcs11.GetSlotList(true)[0]; // Open RW session using (Session session = slot.OpenSession(false)) { // Login as normal user session.Login(CKU.CKU_USER, "11111111"); // Define attributes of RSA public key List<ObjectAttribute> publicKeyAttributes = new List<ObjectAttribute>(); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_CLASS, CKO.CKO_PUBLIC_KEY)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_KEY_TYPE, CKK.CKK_RSA)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_TOKEN, true)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, false)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, "WrapTest")); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ID, ConvertUtils.HexStringToBytes("00010203040506070809"))); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ENCRYPT, true)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY, true)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY_RECOVER, true)); publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_WRAP, true)); // Use modulus extracted from RSA public key publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_MODULUS, ConvertUtils.HexStringToBytes("c3968577a4f007485e77cdf20beca6a4dd200a3d9e478e14db66a4e40534b530c08b1f0604e7dc44e4171b85d84a550b189f94751d7cb048040a66440698f33d8f4634a3e5623a6e69b98563e622df187429738ea4c5e697f236d2d80792803cfc783670ce9697b380cf3603efca098b0db2eac3b48ff80161ea3dd00c7657f7366fc2bafa4ef617ee1a927eff71dcc3037df5ed09bd82dd976be3fd0d192b7d18aac71ff3d7b760946963786558584b597fce913cd586da5e854b8264e708f0e52de82e37f838d7106c876b9750946af38d44ee4ff8f984e168557a83814fa4c2acaca413a7cbc0249bf0b76a2ce1ff2ab9a43463c3be8ede6a4579a6d4168f"))); // Use exponent extracted from RSA public key publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PUBLIC_EXPONENT, ConvertUtils.HexStringToBytes("010001"))); // Import public key ObjectHandle pubKey = session.CreateObject(publicKeyAttributes); // Define attributes of DES key List<ObjectAttribute> desKeyAttributes = new List<ObjectAttribute>(); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_CLASS, CKO.CKO_SECRET_KEY)); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_KEY_TYPE, CKK.CKK_DES3)); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ENCRYPT, true)); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_DECRYPT, true)); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_DERIVE, true)); desKeyAttributes.Add(new ObjectAttribute(CKA.CKA_EXTRACTABLE, true)); // Generate DES key ObjectHandle desKey = session.GenerateKey(new Mechanism(CKM.CKM_DES3_KEY_GEN), desKeyAttributes); // Read value of DES key desKeyAttributes = session.GetAttributeValue(desKey, new List<CKA>() { CKA.CKA_VALUE }); byte[] desKeyValue = desKeyAttributes[0].GetValueAsByteArray(); System.IO.File.WriteAllBytes(@"des.key", desKeyValue); // Wrap key byte[] wrappedKey = session.WrapKey(new Mechanism(CKM.CKM_RSA_PKCS), pubKey, desKey); System.IO.File.WriteAllBytes(@"wrapped.key", wrappedKey); session.DestroyObject(pubKey); session.DestroyObject(desKey); session.Logout(); } }
Unwrap DES3 key with RSA private key:
openssl rsautl -decrypt -pkcs -inkey private.key -in wrapped.key -out unwrapped.key
There is exactly same content/key stored in "des.key" and "unwrapped.key" files. This indicates successful unwrapping.
Alternatively use OpenSSL to wrap DES3 key with RSA public key:
openssl rsautl -encrypt -pkcs -pubin -inkey public.key -in unwrapped.key -out wrapped2.key