Passing byte array to PInvoke call changes it to null

Question

After migration from Visual Studio 2012 to 2013 some PInvoke calls not working as previously.

For example, I'm struggling with this code:

Signature:

 [DllImport(LzoDll64Bit)]
    private static extern int lzo1x_decompress(byte[] src, int src_len, byte[] dst, ref int dst_len, byte[] wrkmem);

Usage:

byte[] dst = new byte[origlen];
int outlen = origlen;
if (Is64Bit())
    lzo1x_decompress(src, src.Length - 4, dst, ref outlen, _workMemory);
else
    lzo1x_decompress32(src, src.Length - 4, dst, ref outlen, _workMemory);

It is expected to lzo1x_decompress(...) to fill newly initialized byte[] dst array, but in VS 2013 the strange behaviour is that after calling that function, dst array turns to null value instead of be filled.

In addition the whole application state seems to be stable and no errors occurring during this.

What may cause this situation or how to avoid this or even debug what is wrong?

Answer 1

You would appear to be going about this the wrong way. It's a worrying sign to see that you switch behaviour depending on whether or not the code is 32 or 64 bit. That is a clear sign that you've got something wrong. The unmanaged declaration looks like this, I believe:

int lzo1x_decompress(const unsigned char *in, size_t in_len,
    unsigned char *out, size_t *out_len, unsigned char *wrkmem);

The matching p/invoke declaration is, for both 32 and 64 bit code, is:

[DllImport(LzoDll, CallingConvention=CallingConvention.Cdecl)
static extern int lzo1x_decompress(byte[] in, IntPtr in_len, 
    [Out] byte[] out, ref IntPtr out_len, byte[] wrkmem);

Your code is erroneously using int for the two size_t parameters. Now, size_t is pointer sized on Windows and so the matching type is IntPtr.