Marshalling Complex structures between C#/C++

Question 1

Okay, I have a working sample. I'm posting this as another answer because it's a very different approach.

So, on the C++ side, I've got this header file:

struct Struct1
{
  int d1[10];
  int d2[10];
};

extern "C" __declspec(dllexport) void __stdcall 
  TestApi2(int* pLength, Struct1 **pStructures);

And the following code:

__declspec(dllexport) void __stdcall 
  TestApi2(int* pLength, Struct1 **pStructures)
{
  int len = 10;

  *pLength = len;
  *pStructures = (Struct1*)LocalAlloc(0, len * sizeof(Struct1));

  Struct1 *pCur = *pStructures;

  for (int i = 0; i < len; i++)
  {
    for (int idx = 0; idx < 10; ++idx)
    {
      pCur->d1[idx] = i + idx;
      pCur->d2[idx] = i + idx;
    }

    pCur ++;
  }
}

Now, the important bit is LocalAlloc. That's actually the place where I've had all the issues, since I allocated the memory wrong. LocalAlloc is the method .NET calls when it does Marshal.AllocHGlobal, which is very handy, since that means we can use the memory in the caller and dispose of it as needed.

Now, this method allows you to return an arbitrary length array of structures. The same approach can be used to eg. return a structure of an array of structures, you're just going deeper. The key is the LocalAlloc - that's memory you can easily access using the Marshal class, and it's memory that isn't thrown away.

The reason you have to also return the length of the array is because there's no way to know how much data you're "returning" otherwise. This is a common "problem" in unmanaged code, and if you've ever done any P/Invoking, you know everything about this.

And now, the C# side. I've tried hard to do this in a nice way, but the problem is that arrays of structures simply aren't blittable, which means you can't simply use MarshalAs(UnmanagedType.LPArray, ...). So, we have to go the IntPtr way.

The definitions look like this:

[StructLayout(LayoutKind.Sequential)]
public class Struct1
{
  [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
  public int[] d1;

  [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
  public int[] d2;
}        

[DllImport(@"Win32Project.dll", CallingConvention = CallingConvention.StdCall)]
static extern void TestApi2(out int length, out IntPtr structs);

Basically, we get a pointer to the length of the "array", and the pointer to the pointer to the first element of the array. That's all we need to read the data.

The code follows:

int length;
IntPtr pStructs;

TestApi2(out length, out pStructs);

// Prepare the C#-side array to copy the data to
Struct1[] structs = new Struct1[length];

IntPtr current = pStructs;
for (int i = 0; i < length; i++)
{
  // Create a new struct and copy the unmanaged one to it
  structs[i] = new Struct1();
  Marshal.PtrToStructure(current, structs[i]);

  // Clean-up
  Marshal.DestroyStructure(current, typeof(Struct1));

  // And move to the next structure in the array
  current = (IntPtr)((long)current + Marshal.SizeOf(structs[i]));
}

// And finally, dispose of the whole block of unmanaged memory.
Marshal.FreeHGlobal(pStructs);

The only thing that changes if you want to really return a structure of an array of structures is that the method parameters move into the "wrapping" structure. The handy thing is that .NET can automatically handle the marshalling of the wrapper, the less-handy thing is that it can't handle the inner array, so you again have to use length + IntPtr to manage this manually.

Question 2

First, don't use unknown-length arrays at all. You're killing any possible use of arrays between managed and unmanaged code - you can't tell the required size (Marshal.SizeOf is useless), you have to manually pass the array length etc. If it's not a fixed length array, use IntPtr:

[StructLayout(LayoutKind.Sequential,Pack=8)]
public struct Struct1
{
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
    public double[] d1;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10)]
    public double[] d2;
}

[StructLayout(LayoutKind.Sequential)]
public struct TestStructOuter
{
     public int length;
     public IntPtr embedded;
}

(if your array of embedded is fixed length, feel free to ignore this, but do include the ByValArray, SizeConst bit. You also have to add ArraySubType=System.Runtime.InteropServices.UnmanagedType.Struct to the attribute).

As you've probably noticed, the TestStructOuter is pretty much useless in this case, it only adds complexity (and do note that marshalling is one of the most expensive operations compared to native languages, so if you're calling this often, it's probably a good idea to keep things simple).

Now, you're only allocating memory for the outer struct. Even in your code, Marshal.SizeOf has no idea how big the struct should be; with IntPtr, this is even more so (or, to be more precise, you're only requesting a single IntPtr, ie. 4-8 bytes or so + the length). Most often, you'd want to pass the IntPtr directly, rather than doing this kind of wrapping (using the same thing in C++/CLI is a different thing entirely, although for your case this is unnecessary).

The signature for your method will be something like this:

[DllImport(@"C:\CPP_Projects\DLL\DLLSample\Release\DLLSample.dll", 
 CallingConvention=System.Runtime.InteropServices.CallingConvention.StdCall)]
public static extern  void testAPI2(ref TestStructOuter pTestStructOuter) ;

Now, if you're going with the IntPtr approach, you want to keep allocating memory manually, only you have to do it properly, eg.:

TestStructOuter outer = new TestStructOuter();
testStructOuter.length = 1;
testStructOuter.embedded = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(Struct1)));

Marshal.StructureToPtr(embedded, testStructOuter.embedded, false);

And finally, you call the method:

// The marshalling is done automatically
testAPI2(ref outer);

Don't forget to release the memory, ideally in the finally clause of a try around everything since the memory allocation:

Marshal.FreeHGlobal(outer.embedded);

Now, this is overly complicated and not exactly optimal. Leaving out the TestStructOuter is one possibility, then you can simply pass the length and pointer to the embedded array directly, avoiding one unnecessary marshalling. Another option would be to use a fixed size array (if it needs to be an array at all :)) in TestStructOuter, which will cure a lot of your headaches and eliminate any need for manual marshalling. In other words, if TestStructOuter is defined as I've noted before:

[StructLayout(LayoutKind.Sequential)]
public struct TestStructOuter
{
     [MarshalAs(UnmanagedType.ByValArray, SizeConst = 10, 
      ArraySubType=UnmanagedType.Struct)]
     public Struct1[] embedded;
}

Suddenly, your whole call and everything becomes as simple as

testAPI2(ref outer);

The whole marshalling is done automatically, no need for manual allocations or conversions.

Hope this helps :)

Question 3

Hint: Leadn C++/CLI. I had to deal with complex interop two times in my life - once with ISDN (AVM devkits make it a lot easier - sadly C++, I needed C#) and now with Nanex (great real time complex and full market ata feeds, sadl complex C++, I need C#)

Both cases I make my own wrapper in C++/CLI, talking down to C++ and exposing a real object model in C#. Allows me to make things a lot nicer and handle a lot of edge cases that our friendly Marshaller can not handle efficiently.