Interfacing with C/C++ routine in C# causes rip

Question

I have a vendor's API call, which I want to use in my C# Windows Forms application. The problem is that no matter what P/Invoke I try, I get a crash on the line stating that the call has unbalanced the stack. I tried working with the vendor, but they do not have Visual Studio 2012 nor have any experience with C# or .NET and after several back and forth emails, they basically washed their hands of the problem.

Here is their official C/C++ declaration

const char *<dll name here>(int Param1,
                            const char *Param2,
                            const char *Param3,
                            unsigned long Param4,
                            unsigned short Param5,
                            unsigned short Param6,
                            unsigned short Param7,
                            unsigned short Param8,
                            unsigned short Param9);

Here is a recent P/Invoke DLLImport line that I tried.

[System.Runtime.InteropServices.DllImport(
    "<dll name here>", EntryPoint = "<AnsiFunctionName>", ExactSpelling = true,
    CharSet = System.Runtime.InteropServices.CharSet.Ansi, SetLastError = true)]
public static extern String <AnsiFunctionName>(int Param1,
                                               String Param2,
                                               String Param3,
                                               ulong Param4,
                                               Int16 Param5,
                                               Int16 Param6,
                                               Int16 Param7,
                                               Int16 Param8,
                                               Int16 Param9);

Here is the actual error message:

Subject Line: PInvokeStackImbalance was detected

A call to P/Invoke function '' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the P/Invoke signature match the target unmanaged signature.

I tried various things, even putting a void return type. I tried not using the return type. I tried instead of Strings, StringBuilders and IntPtrs. No matter what I tried, nothing works.

I mentioned to the vendor after reading an article here on Stack Overflow that returning a string is dangerous and should never be done. Their response was:

"The string return problems mentioned on that page don't apply to our functions. They return a pointer to a statically-allocated string, so you don't need to free it (and must not try to do so). It looks like the "UnmanagedType.LPStr" should work (though the IntPtr thing should work too, with the proper conversions or casting).

Update #1: As mentioned in the comment, I was using uint, not ulong, up until this post. I just thought that I make the comparison look better. Both uint and ulong do not work.

Update #2: Maybe giving the actual function name will help. It is Armadillo CreateCodeShort3A. Going forward, I am using pure .NET, but I do still need to interface to that, just it is not really possible at the moment. More than that, the problem got me curious why I cannot resolve something seemingly so simple as a P/Invoke. After all, how many variations are there? (not a serious question).

Answer 1

Here are the possibilities that I can see:

1. The return value should not be String. Use IntPtr and convert to a string with Marshal.PtrToStringAnsi.
2. The C++ type unsigned long should be matched with uint. You used ulong in C# which is a 64-bit type. In C++ on Windows, unsigned long is an unsigned 32-bit type.
3. The C++ type unsigned short should be matched with ushort.
4. The calling convention used by the C++ DLL is probably cdecl. Add CallingConvention = CallingConvention.Cdecl to your P/Invoke.
5. Don't use SetLastError for such a function. That's used with Win32 API functions that return error conditions through GetLastError. This DLL file almost certainly does not.

So the P/Invoke should look something like this:

[DllImport(@"mydll.dll", CallingConvention=CallingConvention.Cdecl)]
public static extern IntPtr FunctionName(
    int Param1, 
    string Param2, 
    string Param3, 
    uint Param4, 
    ushort Param5, 
    ushort Param6, 
    ushort Param7, 
    ushort Param8, 
    ushort Param9
);