Parallel.For WebRequests - The first requests is just doing nothing

Question

I've got a list of Accounts. I want to login with all accounts on a website. I want to use Parallel.ForEach to process all accounts. This is what my code looks like:

Parallel.ForEach(Accounts,
    acc =>
    {
        acc.WebProxy = null;
        acc.Channel = "pelvicpaladin__";
        Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
        acc.ConnectToIrc().Wait();
    });

Everything works fine except one single problem: The first Account in the Accounts-list does not work. Internal I have to use more than one request (it is a bit more than just logging in). The first request just does nothing. If I break the debugger, there is no available source.

I've got about 12 accounts. I've tried to remove the first account from the list. But the problem is still the same (now the new first (old second) account fails).

And now the very strange point: If I don't use Parallel.For, everything works fine.

foreach (var acc in Accounts)
{
    acc.WebProxy = null;
    Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
    await acc.ConnectToIrc();
}

Again: Everything works except the first account from the list. It is always the first (it does not depend on how much accounts the list contains or which account is the first account).

Does anyone has any idea?

EDIT: This is how I create WebRequests:

private async Task<string> GetResponseContent(HttpWebRequest request)
{
    if (request == null)
        throw new ArgumentNullException("request");

    using (var response = await request.GetResponseAsync())
    {
        return await GetResponseContent((HttpWebResponse)response);
    }
}

private async Task<string> GetResponseContent(HttpWebResponse response)
{
    if (response == null)
        throw new ArgumentNullException("response");

    using (var responseStream = response.GetResponseStream())
    {
        return await new StreamReader(responseStream).ReadToEndAsync();
    }
}

private HttpWebRequest GetRequest(string url)
{
    if (String.IsNullOrWhiteSpace(url))
        throw new ArgumentNullException("url");

    try
    {
        HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
        request.CookieContainer = _cookieContainer;
        request.Referer = url;
        request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
        request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 GTB6 (.NET CLR 3.5.30729)";
        if (_webProxy != null)
            request.Proxy = WebProxy.WebProxy;

        request.KeepAlive = true;
        request.Timeout = 30000;

        return request;
    }
    catch (Exception ex)
    {
        ErrorLogger.Log(String.Format("Could not create Request on {0}.", url), ex);
        return null;
    }
}

Answer 1

You're running into the typical await deadlock situation. The problem is that your calls to ConnectToIrc are using await and capturing the synchronization context. They're trying to marshall the continuations to the main thread. The problem is that your main thread is busy blocking on the call to Parallel.ForEach. It's not allowing any of those continuations to run. The main thread is waiting on the continuations to continue, the continuations are waiting on the main thread to be free to run. Deadlock.

This is (one reason) why you shouldn't be synchronously waiting on asynchronous operations.

Instead just start up all of the asynchronous operations and use WhenAll to wait for them all to finish. There's no need to create new threads, or use the thread pool, etc.

var tasks = new List<Task>();
foreach (var acc in Accounts)
{
    acc.WebProxy = null;
    Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
    tasks.Add(acc.ConnectToIrc());
}
await Task.WhenAll(tasks);

This, unlike your second example, will perform all of the async operations in parallel, while still waiting asynchronously.

Answer 2

Updated again:

    var tasks = Accounts.Select(MyTask).ToList();
    await Task.WhenAll(tasks);

then you can write a named method:

    private Task MyTask(Account acc)
    {
        acc.WebProxy = null;
        Debug.WriteLine("Connecting to {0}.", new object[] { acc.Username });
        return acc.ConnectToIrc();
    }

thanks for the tip