Why does my TPL program use more ThreadPool resources than its async/await counterpart?

Question

I am writing a program that demonstrates the benefits of using asynchronous IO in the context of server scalability. The program concurrently consumes an asynchronous method, and then reports the IDs of the threads that participated in the asynchronous processing.

To illustrate, consider the following:

static async Task<TimeSpan> AsyncCalling(TimeSpan time)
{
    using (SleepService.SleepServiceClient client = new SleepService.SleepServiceClient())
    {
        TimeSpan response = await client.SleepAsync(time);
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);

        response += await client.SleepAsync(TimeSpan.FromTicks(time.Ticks / 2));
        Console.WriteLine(Thread.CurrentThread.ManagedThreadId);

        return response;
    }
}

I simulate a server under load by calling the above async method as follows:

int numberOfWorkItems = 50;
for (int i = 0; i < numberOfWorkItems; ++i)
{
    TimeSpan value = TimeSpan.FromSeconds((i % 3) + 1);
    ThreadPool.QueueUserWorkItem(arg => { TimeSpan t = AsyncCalling(value).Result; });

    Thread.Sleep(300);
}

The ThreadPool.QueueUserWorkItem operation simulates the allocation of a request thread, with the AsyncCalling method being the method executed by the request (similar to the operation of WCF).

Execution is as expected, and I only count two or three distinct thread IDs when analyzing the output. This is typical for my machine as I only have two cores and the thread pool will resist scheduling more threads than cores available.

Now I attempt to do the same analysis, but for a TPL function that doesn't use the await keyword. The function is as follows:

static Task<TimeSpan> TaskAsyncCalling(TimeSpan time)
{
    SleepService.SleepServiceClient client = new SleepService.SleepServiceClient();

    return client.SleepAsync(time)
                 .ContinueWith(t =>
                 {
                     TimeSpan result = t.Result;
                     Console.WriteLine(Thread.CurrentThread.ManagedThreadId);

                     return client.SleepAsync(TimeSpan.FromTicks(time.Ticks / 2))
                                  .ContinueWith(t1 =>
                                  {
                                      result += t1.Result;
                                      Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
                                      (client as IDisposable).Dispose();

                                      return result;
                                  });
                 })
                 .Unwrap();
}

When TaskAsyncCalling is called in the same context, the output results are completely different. The tasks generally take much longer to execute, and total number of unique thread IDs is generally on the order of 30 (again, for my 2-core machine).

Why does this disparity exist? I understand that await is not a trivial wrapper over Task<T>, however the thread pool is the common denominator and I expected the same clever thread reuse to occur in the TPL implementation.

Is there another way to rewrite the TPL method to achieve the same results without blocking?

---

Edit:

The SleepAsync call is the asynchronous generated WCF client method for the following synchronous operation. Note that in this case, the client does not block where as the server does.

public TimeSpan Sleep(TimeSpan time)
{
    Thread.Sleep(time);
    return time;
}

Answer 1

In this case, the classic TPL version uses more threads than the async/await version because each ContinueWith continuation is executed on a separate pool thread.

Fix that with TaskContinuationsOptions.ExecuteSynchronously:

static Task<TimeSpan> TaskAsyncCalling(TimeSpan time)
{
    SleepService.SleepServiceClient client = new SleepService.SleepServiceClient();

    return client.SleepAsync(time)
        .ContinueWith(t =>
        {
            TimeSpan result = t.Result;
            Console.WriteLine(Thread.CurrentThread.ManagedThreadId);

            return client.SleepAsync(TimeSpan.FromTicks(time.Ticks / 2))
                .ContinueWith(t1 =>
                {
                    result += t1.Result;
                    Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
                    (client as IDisposable).Dispose();

                    return result;
                }, TaskContinuationsOptions.ExecuteSynchronously);
         }, TaskContinuationsOptions.ExecuteSynchronously)
         .Unwrap();
 }

OTOH, the await continuations are normally executed synchronously (if the operation completed on the same synchronization context it was started on, or if there was no synchronization at both points of execution). So it's expected to acquire two less threads.

A good related read: "Why is TaskContinuationsOptions.ExecuteSynchronously opt-in?"

Answer 2

I am not sure that the two implementations are the same, await stores the current SyncrhonizationContext and gets the TaskScheduler associated with that. Which is not what the default implementation of ContinueWith does. From reflector:

public Task ContinueWith(Action<Task<TResult>> continuationAction)
{
    StackCrawlMark lookForMyCaller = StackCrawlMark.LookForMyCaller;
    return this.ContinueWith(continuationAction, TaskScheduler.Current, new CancellationToken(), TaskContinuationOptions.None, ref lookForMyCaller);
}

So ContinueWith uses the TaskScheduler.Current whereas await uses the TaskScheduler associated with the current SyncrhonizationContext. If the two are not the same you may get different behavior.

Try specifying a TaskScheduler.FromCurrentSynchronizationContext() for ContinueWith and see if there is any difference.