Why can you not use yield in a lambda, when you can use await in a lambda?

Question 1

According to Eric Lippert, anonymous iterators were not added to the language because it would be overly complicated to implement it.

That is not precisely what I intended to convey. The relevant cost is implementation cost, yes, but it is implementation cost in an existing compiler which was not set up architecturally to implement that complex feature.

The compiler has to do the same thing for async methods as it has to for iterators (convert them into state machines), so I am very confused why anonymous iterators are not allowed as well, when anonymous async methods are.

A brief history is relevant. C# first had anonymous methods and iterator blocks in C# 2.0. When I added lambdas in C# 3.0 it was a major cost to refactor all of the existing anonymous method code so that it could handle all the new features of lambdas. That made it even more complicated and expensive to modify. Making iterator block lambdas was judged too costly for the benefits that would be accrued; it would have been a large percentage of the total cost. We could not afford it. If you added up every team in Developer Division's work schedule, the team with the "longest pole" was the C# 3.0 compiler team, and my work on the semantic analyzer was IIRC the longest pole on the compiler team. Every day we might have slipped C# 3.0, that would have been a day that Visual Studio would have slipped. Therefore anything that didn't make LINQ better was cut, and that included iterator lambdas.

In C# 4, iterator lambdas were one feature of many that were considered. We had a list of potential good features literally longer than your arm and we could afford to do less than a tenth of them.

In C# 5 the team added async methods. The design and implementation teams tried for a long time to come up with an underlying abstraction that was common to both the iterator block and await rewrites; they are obviously similar, as you note. But ultimately, the cost of finding the general solution did not pay for itself. Generality is surprisingly expensive, and finding a generality that by design unifies only two things is silly if it is not cheap.

Therefore the decision was made to implement the await rewriter as its own thing. Given that the team was going to take on this large cost, and given that the original transformation of async methods was going to be into a lambda form anyway, the decision was made to invest in the full feature: async methods containing lambdas, async lambdas containing lambdas, the whole deal. The cost of that feature was a small fraction of the cost of the whole feature, which was extremely expensive.

And again, we have a problem with long poles. Any work on the lambda engine that could potentially have destabilized await is to be avoided, and that includes trying to make them work with iterator blocks.

Now compare Visual Basic. VB for a long time had no iterator blocks at all. When they were added, there was no existing infrastructure to keep working! The whole thing could be built from the ground up to handle iterator blocks containing lambdas and lambdas containing iterator blocks, and so that was done.

The C# compiler has been thoroughly rearchitected and rewritten via the Roslyn project. I am hoping that this will lower the cost of implementing iterator block lambdas in a hypothetical future version of C#. We shall see!

Question 2

Anonymous iterator blocks, while nice, do not have a particularly compelling benefit. It is not a huge deterrent for iterator blocks to be refactored into their own method.

async anonymous methods make a lot more conceptual sense, don't warrant refactoring into their own method quite the same way anonymous iterator blocks do, and have a much more compelling end user benefit.

In short, the benefits were worth the cost to implement, unlike iterator blocks. The costs were likely rather comparable.

Question 3

Look at this code (it does not work, just an example):

Task<IEnumerable<int>> resultTask = new Task<IEnumerable<int>>(() =>
{
    for (int i = 0; i < 10; ++i)
    {
        yield return i;
    }
});

Don't you find it some kind of unstructured?

Assuming whole range of lambdas usages, it would be so hard and not worth it to handle yield "laziness" properly.

However, there are great approaches to yield return from parallel tasks.

But let's have a look on following thing. Defining a method with yield return:

static IEnumerable<int> GetIntegers()
{
    for (int i = 0; i < 10; ++i)
    {
        yield return i;
    }
}

And putting it in lambda will work:

Task<IEnumerable<int>> resultTask = new Task<IEnumerable<int>>(() =>
{
    return GetIntegers();
});

What is the way this code will behave in? Is it going to lose real yield advantages?