Parallel.ForEach
is good for CPU-bound computational tasks, but it will unnecessary block pool threads for synchronous IO-bound calls like DownloadString
in your case. You can improve the scalability of your code and reduce the number of threads it may use, by using DownloadStringTaskAsync
and tasks instead:
// non-blocking async method
async Task<string> ProcessUrlAsync(string url)
{
using (var webClient = new WebClient())
{
string data = await webClient.DownloadStringTaskAsync(new Uri(url));
// run checks here..
return data;
}
}
// ...
if (ListOfUrls.Count > 0) {
var tasks = new List<Task>();
foreach (var url in ListOfUrls)
{
tasks.Add(ProcessUrlAsync(url));
}
Task.WaitAll(tasks.ToArray()); // blocking wait
// could use await here and make this method async:
// await Task.WhenAll(tasks.ToArray());
}