How do I work around the error "ByRef return value not supported in reflection invocation" in C#?

Question

I have a .Net library supplied by a third party. I did reflection on one of their classes and found a member method. The signature was...

Byte& FooBar()

So, I wanted to call this method through reflection and got the exception "ByRef return value not supported in reflection invocation."

Here is what I've tried...

        var strm = new TheirClass();
        var t = strm.GetType();
        var ms = t.GetMembers(
                    BindingFlags.Static|BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
        foreach (var m in ms)
        {
            Debug.WriteLine(String.Format("Name: {0}: {1}", m.Name, m.ToString()));
            // ...
            // Name: FooBar:  Byte& FooBar()
            // ...
        }
        var meth = t.GetMethod("FooBar");
        object returnValue = meth.Invoke(strm, new object[] {  });  //throw exception

I have tried supplying parameters as in calling functions with ref parameters, but that made no difference.

I would like to work around this exception in C#.

Answer 1

Per the comments: here is how it can be done from CIL, which can be generated from C#.

I was hoping to use a DynamicMethod, but I cannot manage to get this working without creating a custom delegate type at runtime, so I needed to use AssemblyBuilder instead.

using System;
using System.Reflection;
using System.Reflection.Emit;

public delegate void CallBadFunction(Delegate d, Callback c);
public delegate void Callback(ref int i);

static class Program
{
    static int i;
    static object BadMethod()
    {
        return i;
    }

    static MethodInfo GetBadMethod()
    {
        return typeof(Program).GetMethod("BadMethod", BindingFlags.Static | BindingFlags.NonPublic);
    }

    static void Main()
    {
        var badMethod = GetBadMethod();

        var assembly = AssemblyBuilder.DefineDynamicAssembly(new AssemblyName("-"), AssemblyBuilderAccess.Run);
        var module = assembly.DefineDynamicModule("-");

        var badDelegate = module.DefineType("BadDelegateType", TypeAttributes.Public | TypeAttributes.Class | TypeAttributes.Sealed, typeof(MulticastDelegate));
        var badDelegateCtor = badDelegate.DefineConstructor(MethodAttributes.Public | MethodAttributes.SpecialName | MethodAttributes.RTSpecialName, CallingConventions.Standard, new Type[] { typeof(object), typeof(IntPtr) });
        badDelegateCtor.SetImplementationFlags(MethodImplAttributes.Runtime | MethodImplAttributes.Managed);
        var badDelegateInvoke = badDelegate.DefineMethod("Invoke", MethodAttributes.Public | MethodAttributes.Virtual | MethodAttributes.NewSlot | MethodAttributes.HideBySig, typeof(int).MakeByRefType(), Type.EmptyTypes);
        badDelegateInvoke.SetImplementationFlags(MethodImplAttributes.Runtime | MethodImplAttributes.Managed);
        var badDelegateType = badDelegate.CreateType();

        var method = module.DefineGlobalMethod("-", MethodAttributes.Public | MethodAttributes.Static, typeof(void), new[] { typeof(Delegate), typeof(Callback) });
        var il = method.GetILGenerator();
        il.Emit(OpCodes.Ldarg_1);
        il.Emit(OpCodes.Ldarg_0);
        il.Emit(OpCodes.Castclass, badDelegate);
        il.Emit(OpCodes.Callvirt, badDelegateInvoke);
        il.Emit(OpCodes.Callvirt, typeof(Callback).GetMethod("Invoke"));
        il.Emit(OpCodes.Ret);
        module.CreateGlobalFunctions();

        var callBadFunction = (CallBadFunction)Delegate.CreateDelegate(typeof(CallBadFunction), module.GetMethod("-"));
        callBadFunction(badMethod.CreateDelegate(badDelegateType), (ref int i) =>
        {
            i++;
        });
    }
}

After compiling this program, use ILDASM to disassemble it, and replace BadMethod's definition by

.method private hidebysig static int32&
        BadMethod() cil managed
{
  ldsflda     int32 Program::i
  ret
}

This turns it into a function returning int32&, which the following code will then manage to call. The only location C# allows int32& types is in function parameters (ref int), so to make the result usable, I used a callback function, which gets passed the return value of BadMethod.

Answer 2

Thank you "hvd" for pointing me in the right direction. After some thought, here is what I came up with. It's simpler, but please tell me if you see a flaw.

        var theirObj = new TheirClass();
        var t = theirObj.GetType();
        var fooBar = t.GetMethod("FooBar"); // Byte& FooBar()

        DynamicMethod dm = new DynamicMethod(
            "MyFooBar",
            MethodAttributes.Public | MethodAttributes.Static,
            CallingConventions.Standard,
            typeof(IntPtr), 
            new Type[] { typeof(TheirClass) },
            typeof(TheirClass),
            true);

        var ilg = dm.GetILGenerator();
        ilg.Emit(OpCodes.Ldarg_0);
        ilg.Emit(OpCodes.Call, foobar);
        ilg.Emit(OpCodes.Ret);

        var del = dm.CreateDelegate(typeof(Func<TheirClass,IntPtr>));
        var ret = (IntPtr)del.DynamicInvoke(theirObject);

        byte[] buf = new byte[theirObj.FooBarSize()]; //no need for reflection/IL here
        // not sure about the following, it works, but should it be inside an "unsafe" section?
        Marshal.Copy(ret, buf, 0, buf.Length);

I put a simple wrapper around the offending method and IL doesn't care that I treat the Byte& as an IntPtr. I'm still not sure about doing a Copy without an unsafe wrapper. But this is good for now.