Requests canceled with server under heavy load hang on IIS

Question

I am facing a difficult-to-reproduce problem that I think is related to the use of CancellationToken. I have three async apps (each on a different server) that communicate through ASP.NET Web API as shown in the diagram below

A ---> B ---> C

App A runs a routine from time to time that fires a lot of concurrent requests to app B, which in turn fires another lot of concurrent requests to app C. All these calls are done using HttpClient.SendAsync, and are constrained to a 10 seconds timeout, using a CancellationToken as such:

var cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(10));
var token = cancellationTokenSource.Token;

When server C is under heavy load, it starts taking more time to respond, which causes server B to start cancelling requests -- because of the configured Timeout. When this happens, sometimes a request gets "stuck" (see image below).

I am familiarized with C# async best practices, and I am not doing .Result or .Wait() or the like. I am aware that these can cause hangs.

If I pass CancellationToken.None instead, the problem goes away, so my hint is that it has to do with canceling under high concurrency scenario.

When time passes, more and more requests start hanging, and eventually I have no choice but to do an IIS reset to make them go away.

Any clues of what can be going on?

Answer 1

cancellationtoken doesnt dictates cancelation, implementer may inspect it after very long interval, or ignore it completely. the correct way implementing timeout is:

if(yourTask == Task.WhenAny(yourTask, Task.Delay(3000)))
{
   //task completed
}
else
{
  //timeout occured
}