How to P/Invoke a function that allocates and returns pointer to new array

Question

I have a C function that I would like to call from C# program. Function compresses inputted byte array and outputs new compressed array. It looks like this:

extern __declspec(dllexport) int Compress(chandle handle, 
unsigned char *inputBuf, unsigned char **outputBuf, unsigned long *outputSize);

I have translated it into this on C# part. But output I get is array with one item.

[DllImport("compresslib.dll", CallingConvention = CallingConvention.Cdecl)]
internal extern static int Compress(IntPtr handle, byte[] input, out byte[] output, out uint outputSize);

What should I do to get this working?

Here is working code I was able to write with the help of Hans Passant

[DllImport("compresslib.dll", CallingConvention = CallingConvention.Cdecl)]
internal extern static int Compress(IntPtr handle, byte[] input, out IntPtr output, out uint outputSize);

// and this is how i call it 
byte[] outputData;
int outputDataSize;
IntPtr outputDataP = IntPtr.Zero;
try
{
    int success = NativeMethods.Compress(handle,
    inputData, out outputDataP, out outputDataSize);
    if (success == -1)
    {
        throw new Exception("Compression failed.");
    }
    outputData = new byte[outputDataSize];
    Marshal.Copy(outputDataP , outputData , 0, (int)outputDataSize);
}
finally
{
    if (outputDataP != IntPtr.Zero)
    NativeMethods.tjFree(outputDataP);// release unmanaged buffer
}

return outputData ;

Answer 1

  ..., unsigned char **outputBuf, ...

There is a pretty serious problem with this function, it cannot reliably be called from a C program either. The caller needs to release the output buffer after using it. That requires using the exact same allocator as was used in your C code. That's very hard to guarantee in a C program and often goes wrong badly. Like when the user of your DLL doesn't use the exact same compiler version as you used. He'll use a different version of the C runtime library, one that uses his own heap. So cannot possibly release the buffer since he doesn't have a handle to the heap that you used.

It becomes next to impossible when you pinvoke, the CLR of course has absolutely no idea what C runtime version you used and is guaranteed to not use the same one that you used since it has its own private copy.

The reason you are just getting one byte is related to this problem, the pinvoke marshaller doesn't know how large the array is since it didn't create the array. That's fixable you need to apply tbe [MarshalAs(UnmanagedType.LPArray), SizeParamIndex = 3] attribute to the parameter. That tells the pinvoke marshaller that the 4th argument contains the size of the array.

You'll need to address the memory management problem, you can't leave it the way it is since you'll leak memory badly on XP and hard-crash on Vista and higher when the pinvoke marshaller tries to release the array. You need to alter the C code to use memory for the returned output buffer that is allocated on a known heap. Which requires using CoTaskMemAlloc().

Another possible fix is that you export a function that allows the caller to release the buffer. In which case you should declare the argument out IntPtr and marshal yourself with, say, Marshal.Copy(), then release the buffer with your added function.

Or a completely different way to use the function, like having Compress() only compress but not return the data yet. With extra functions that the client code can use to discover the required buffer size and to obtain a copy of the data.