Difficult Marshalling DirectX array of structs from C++ to C#

Question

Goal: Marshal C++ (pointer to an?) array of structs to C#.

C++: CreateVertexDeclaration()

   HRESULT CreateVertexDeclaration(
     [in]           const D3DVERTEXELEMENT9 *pVertexElements,
     [out, retval]  IDirect3DVertexDeclaration9 **ppDecl
   );

C#:

I'm using this to define the D3DVERTEXELEMENT9 structure. SharpDX is a managed DirectX library generated directly from the DirectX SDK C++ headers, so it's supposedly quite compatible for COM interop. I've actually already got 10 other hooks working perfectly, but this is the first with a pointer to an array of structures.

Just in case SharpDX's structure definition doesn't work, I've also tried replacing it with my own definition:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
    public class D3DVERTEXELEMENT9
    {
        public short Stream;
        public short Offset;
        public DeclarationType Type;      // Enum
        public DeclarationMethod Method;  // Enum
        public DeclarationUsage Usage;    // Enum
        public byte UsageIndex;
    }

I have tried the following C# method signatures:

Note: Having IntPtr devicePointer as the first parameter shouldn't be the problem. I have it for 10 other hooks that works successfully.

Note: These are Direct3D API hooks. So my program is being passed the data that I need to marshal from an unmanaged C++ environment to a managed C# environment.

1:

CreateVertexDeclaration(IntPtr devicePointer, D3DVERTEXELEMENT9[] vertexElements, out IntPtr vertexDeclaration)

Result: The array has one element, but its values are default (doesn't make sense). This means that short Stream, short Offset, Type, Method, Usage, and UsageIndex are all 0 (the enums take their first value).

2:

CreateVertexDeclaration(IntPtr devicePointer, ref D3DVERTEXELEMENT9[] vertexElements, out IntPtr vertexDeclaration)

Result: The array itself is null.

3:

CreateVertexDeclaration(IntPtr devicePointer, [In, Out] D3DVERTEXELEMENT9[] vertexElements, out IntPtr vertexDeclaration)

Same as 1. No benefit.

4:

CreateVertexDeclaration(IntPtr devicePointer, D3DVERTEXELEMENT9 vertexElements, out IntPtr vertexDeclaration)

Same as 1, doesn't change anything. I get a defaulted structure. The same if I called new D3DVERTEXELEMENT9().

5:

CreateVertexDeclaration(IntPtr devicePointer, ref D3DVERTEXELEMENT9 vertexElements, out IntPtr vertexDeclaration)

I forgot the result, but it didn't work. I think it actually crashed the hooks.

6:

CreateVertexDeclaration(IntPtr devicePointer, IntPtr vertexElements, out IntPtr vertexDeclaration)

I think either this #6 or #1/#2 are supposed to be correct. I've probably messed up the implementation for this method?

I tried to use this code with it:

var vertex = (D3DVERTEXELEMENT9)Marshal.PtrToStructure(vertexElements, typeof(D3DVERTEXELEMENT9));

But it doesn't work! It's the same as #1. My marshalled pointer-to-structure is a defaulted structure, the same as if I called new D3DVERTEXELEMENT9().

7:

CreateVertexDeclaration(IntPtr devicePointer, ref IntPtr vertexElements, out IntPtr vertexDeclaration)

Null or zero; not valid.

---

For method #6 (using IntPtr), I tried viewing the memory region in Visual Studio. The next 50 bytes or so were all zeros. There was maybe one 2 byte in the field of 50 zeros. But it was pretty much 49 bytes of zeros starting from the IntPtr supplied to the method.

Isn't that a problem? Doesn't that mean that the supplied pointer is already incorrect? So I took that to mean I had the wrong method signature. The problem is I've been trying all sorts of possible combinations, and they either crash the program or give me a default array the same as if I called new D3DVERTEXELEMENT9().

Question: What is the solution to correctly marshalling the pVertexElements array of structs from C++ to C#, and why are my current signatures incorrect?

Additional Note: In C++, the length of this specific array is defined by suffixing a D3DDECL_END. I have no idea how I'm supposed to get the corresponding array length in C# without some sort of passed length parameter.

---

Other Working Hooks:

C++:

BeginScene(LPDIRECT3DDEVICE9 pDevice)
BeginStateBlock(LPDIRECT3DDEVICE9 pDevice)
Clear(LPDIRECT3DDEVICE9 pDevice, DWORD Count,CONST D3DRECT* pRects,DWORD Flags,D3DCOLOR Color,float Z,DWORD Stencil)
ColorFill(LPDIRECT3DDEVICE9 pDevice, IDirect3DSurface9* pSurface,CONST RECT* pRect,D3DCOLOR color)
CreateAdditionalSwapChain(LPDIRECT3DDEVICE9 pDevice, D3DPRESENT_PARAMETERS* pPresentationParameters,IDirect3DSwapChain9** pSwapChain)
CreateCubeTexture(LPDIRECT3DDEVICE9 pDevice, UINT EdgeLength,UINT Levels,DWORD Usage,D3DFORMAT Format,D3DPOOL Pool,IDirect3DCubeTexture9** ppCubeTexture,HANDLE* pSharedHandle)

...

C#:

Note: I use SharpDX enums and structures for all of these delegates, and they work fine. They also all begin with IntPtr devicePointer.

BeginSceneDelegate(IntPtr devicePointer);
BeginStateBlocKDelegate(IntPtr devicePointer);
ClearDelegate(IntPtr devicePointer, int count, IntPtr rects, ClearFlags flags, ColorBGRA color, float z, int stencil);
ColorFillDelegate(IntPtr devicePointer, IntPtr surface, IntPtr rect, ColorBGRA color);
CreateAdditionalSwapChainDelegate(IntPtr devicePointer, [In, Out] PresentParameters presentParameters, out SwapChain swapChain);
CreateCubeTextureDelegate(IntPtr devicePointer, int edgeLength, int levels, Usage usage, Format format, Pool pool, out IntPtr cubeTexture, IntPtr sharedHandle);
...

Log of passed parameters for other hooks:

DLL injection suceeded.
Setting up Direct3D 9 hooks...
Activating Direct3D 9 hooks...
CreateDepthStencilSurface(IntPtr devicePointer: 147414976, Int32 width: 1346, Int32 height: 827, Format format: D24S8, MultisampleType multiSampleType: None, Int32 multiSampleQuality: 0, Boolean discard: False, IntPtr& surface: (out), IntPtr sharedHandle: 0)
CreateDepthStencilSurface(IntPtr devicePointer: 147414976, Int32 width: 1346, Int32 height: 827, Format format: D24S8, MultisampleType multiSampleType: None, Int32 multiSampleQuality: 0, Boolean discard: False, IntPtr& surface: (out), IntPtr sharedHandle: 0)
Clear(IntPtr devicePointer: 147414976, Int32 count: 0, IntPtr rects: (Empty), ClearFlags flags: Target, ColorBGRA color: A:0 R:0 G:0 B:0, Single z: 1, Int32 stencil: 0)
Clear(IntPtr devicePointer: 147414976, Int32 count: 0, IntPtr rects: (Empty), ClearFlags flags: Target, ColorBGRA color: A:0 R:0 G:0 B:0, Single z: 1, Int32 stencil: 0)
BeginScene(IntPtr devicePointer: 147414976)
Clear(IntPtr devicePointer: 147414976, Int32 count: 0, IntPtr rects: (Empty), ClearFlags flags: Target, ColorBGRA color: A:0 R:0 G:0 B:0, Single z: 1, Int32 stencil: 0)
Clear(IntPtr devicePointer: 147414976, Int32 count: 0, IntPtr rects: (Empty), ClearFlags flags: Target, ColorBGRA color: A:0 R:0 G:0 B:0, Single z: 1, Int32 stencil: 0)
Clear(IntPtr devicePointer: 147414976, Int32 count: 0, IntPtr rects: (Empty), ClearFlags flags: ZBuffer, ColorBGRA color: A:0 R:0 G:0 B:0, Single z: 1, Int32 stencil: 0)
BeginScene(IntPtr devicePointer: 147414976)

The method signature most similar to this CreateVertexDeclaration() seems to be Clear(). Here is my implementation of Clear():

private Result Clear(IntPtr devicePointer, int count, IntPtr rects, ClearFlags flags, ColorBGRA color, float z, int stencil)
        {
            try
            {
                var structSize = Marshal.SizeOf(typeof (Rectangle));
                var structs = new Rectangle[count];
                for (int i = 0; i < count; i++)
                {
                    structs[i] = (Rectangle) Marshal.PtrToStructure(rects, typeof (Rectangle));
                }
                // Seems to work fine, not sure why it doesn't work for CreateVertexDeclaration

                var rectangles = structs;
                Log.LogMethodSignatureTypesAndValues(devicePointer, count, rectangles.PrintTypesNamesValues(), flags,
                                                     color, z, stencil);
                GetOrCreateDevice(devicePointer);
                if (rectangles.Length == 0)
                    Device.Clear(flags, color, z, stencil);
                else
                    Device.Clear(flags, color, z, stencil, rectangles);
            }
            catch (Exception ex)
            {
                Log.Warn(ex.ToString());
            }

            return Result.Ok;
        }

Answer 1

First, the declaration in SharpDX is using a Pack = 2 (and not a Pack = 1):

    [StructLayout(LayoutKind.Sequential, Pack = 2 )]
    public  partial struct VertexElement {  
     ...
    }

Also if you want to be sure about the marshal, prefer using unsafe than Marshal.StructureToPtr or any other marshal method whenever it is possible (when the structure maps directly to the C# equivalent). You will avoid Marshal performance issues and you will be sure about the memory layout (assuming that the structure was correctly declared in c#)

public unsafe static void CreateVertexDeclaration(IntPtr devicePointer, IntPtr vertexElementsPtr, out IntPtr vertexDeclaration)
{
    var vertexElements = (VertexElement*)vertexElementsPtr;

    // Access to all VertexElement (last element is specified by Cmacro D3DDECL_END() in d3d9types.h)
    ...
}