PInvoke in 64bit .net app through c++ 64 bit dll

Question

I'm having an issue calling a function in a c++ dll inside of a c# app. I'm calling the function inside of c# like so:

[DllImport("cryptopp.dll")]
public static extern IntPtr RSAEncryptString(string filename, string seed, string message);

It is being exported in the c++ dll as shown below.

extern "C" __declspec(dllexport) const char* __cdecl RSAEncryptString(const char *pubFilename, const char *seed, const char *message);

What I get when I try to call this, however, is an "An External component has thrown an exception." exception, which is not very descriptive at all, and extremely unhelpful.

When I pull up the dll in an export viewer, it shows all the other exported functions with fully quantified declarations (I.E. public: void __cdecl CryptoPP::X509PublicKey::`vbase destructor'(void) __ptr64 ) , except for the function I am calling, which just displays the function name RSAEncryptString.

This is the only possible issue I can see, besides maybe mis-calling the function with an invalid declaration on the c# side. Am I using System.Runtime.InteropServices.Marshal wrong?

Please help <3 and thanks in advance.

Answer 1

I think you need to change the first line to:

[DllImport("cryptopp.dll",
    CharSet = CharSet.Ansi, CallingConvention = CallingConvention.Cdecl)]

If you want to get very descriptive, you can also add these:

public static extern IntPtr RSAEncryptString(
    [In, MarshalAs(UnmanagedType.LPStr)] string filename,
    [In, MarshalAs(UnmanagedType.LPStr)] string seed,
    [In, MarshalAs(UnmanagedType.LPStr)] string message);

IIRC think the CharSet should take care of the encoding thing for you, but if it doesn't, use the MarshalAs also, as shown above.

---

Edit:

Oh I think I got why you still get an error! Your code still had the above problems, but it's still erring because you can't return a string object since it's not a managed object; you need to return a pointer (like IntPtr) and then use Marshal.PtrToStringAnsi!

(I didn't really look at your return type when answering this at first.)

Answer 2

It appears you're trying to store the return value of type const char * (an LPCSTR) into an IntPtr type (usually used for HANDLEs, not LPSTRs.) Try this:

[DllImport("cryptopp.dll", CharSet = CharSet.Auto)]
public static extern String RSAEncryptString(String filename, String seed, String message);

Also keep in mind that if any argument is getting written to, you'll need to add out before its type, i.e. ..., out String message)