DynamicMethods, Expression Trees and the DLR

Question

I have some questions about the interactions and relationships between DynamicMethods, Expression Trees and the DLR.

1. I know that LambdaExpression.Compile uses an ILGenerator internally to create a Delegate. However, there are some fundamental differences between a compiled LambdaExpression and a DynamicMethod. For instance

   a. DynamicMethods invoke faster

   b. Compiled LambdaExpressions can embed closures (ConstantExpressions that are non primitive values)

   b. Compiled LambdaExpressions have no DeclaringType.

   Questions:

   a. Why are DynamicMethods faster to invoke than compiled LambdaExpressions?

   b. What's special about compiled LambdaExpressions that allows for closures? Does the Expression Tree actually generate a closure class when I use a non ConstantExpression? And if so, where does this generated class go?

   c. Where do compiled LambdaExpressions go (at runtime)? Where is the support for them implemented. It can't just be Reflection.Emit, can it?

2. I know that the dynamic keyword is really just a compiler trick for emitting CSharp CallSites, Binders, etc. Internally, as I understand, these generate Expression Trees and also use a stripped down version of the C# compiler.

   Questions

   a. Are the expression trees being generated a function of CallSiteBinders in general or the specific implementation and usage of them in the Microsoft.CSharp dll?

   b. Are these expression trees composed of DynamicExpression nodes? Or something else? If something else, why?

   c. Where and why does the stripped down version of the C# compiler come into play? Why and how is it different than regular calls to LambdaExpression.Compile or DynamicMethods or any kind of IL generation? I can understand how CallSiteBinders could be used to build Expression Trees, but why is the C# compiler needed after that transformation occurs? And what does C# have to do with it at all once it's in the form of an Expression Tree (which is just an API).

Answer 1

I don't know much about dynamic, so I'm going to answer only the first part of your question.

Why are DynamicMethods faster to invoke than compiled LambdaExpressions?

I would be very surprised if they were, since Expression.Compile() internally uses DynamicMethod.

What's special about compiled LambdaExpressions that allows for closures? Does the Expression Tree actually generate a closure class when I use a non ConstantExpression? And if so, where does this generated class go?

That's easy to verify. Just look at Target and Method of a delegate generated from compiling an expression tree. You will notice that the Target (and first parameter of Method) is System.Runtime.CompilerServices.Closure. That's a class that contains a field object[] Constants, which is where non-primitive values from ConstantExpressions are stored.

Where do compiled LambdaExpressions go (at runtime)? Where is the support for them implemented. It can't just be Reflection.Emit, can it?

Like I said before, Expression.Compile() internally uses DynamicMethod. So, yeah, it is just Reflection.Emit.

Answer 2

Well, I can't answer all your questions, but I can answer a few of them and I think that may answer most of yours. Perhaps at the minimum it will give you enough information to continue researching.

Why are DynamicMethods faster to invoke than compiled LambdaExpressions?

I don't think they are, perhaps you measured wrong and it's a JIT'ing difference

What's special about compiled LambdaExpressions that allows for closures? Does the Expression Tree actually generate a closure class when I use a non ConstantExpression? And if so, where does this generated class go?

This one I am unsure about. I would assume that Expression.Constant could contain reference types, then it's a non-issue, but if it can indeed only have value types, then I would guess that the compiler would just be generating an expression where the variables being captured in the closure are just passed in as parameters.

Where do compiled LambdaExpressions go (at runtime)? Where is the support for them implemented. It can't just be Reflection.Emit, can it?

System.Linq.Expressions is really just a friendlier API on top of Reflection.Emit, ergo they are just stored in memory just like Reflection.Emit is by default (though with Reflection.Emit you can save out the emitted code I believe)

Are the expression trees being generated a function of CallSiteBinders in general or the specific implementation and usage of them in the Microsoft.CSharp dll?

I've only done a little System.Dynamic work, so I cannot answer this, but it's been my understanding that the CallSiteBinder just caches and invokes the expression, but passes the actual generation off to something else (ie DynamicObject). But again, you likely know more here than I do.

Are these expression trees composed of DynamicExpression nodes? Or something else? If something else, why?

No, dynamic is still bound by the same rules as everything else in .net. Dynamic is just saying "at runtime, when i do x, go and try to build the code that I would have normally written and execute it for me." Things like DynamicObject are just going to build a plain old expression tree, dynamic object just provides you some meta data so that you can in fact build that tree (like return type, type of access, name, etc).

Where and why does the stripped down version of the C# compiler come into play? Why and how is it different than regular calls to LambdaExpression.Compile or DynamicMethods or any kind of IL generation? I can understand how CallSiteBinders could be used to build Expression Trees, but why is the C# compiler needed after that transformation occurs? And what does C# have to do with it at all once it's in the form of an Expression Tree (which is just an API).

I'm not sure what you mean by stripped down compiler or what the runtime actually does to generate the IL code (which I think is what you are after here), so I don't think I can answer this.

I will say, however, like I said before: the System.Linq.Expression stuff is really just a friendly API on top of Reflection.Emit. The expression tree is just the information it needs to go to reflection.emit, generate a dynamic method, and return it back to you.