This seems like a job for PLINQ:
return source.AsParallel().Select(s => ExecuteOrDownloadSomething(s));
This will execute the delegate in parallel using a limited number of threads, returning each result as soon as it completes.
If the ExecuteOrDownloadSomething()
method is IO-bound (e.g. it actually downloads something) and you don't want to waste threads, then using async
-await
might make sense, but it would be more complicated.
If you want to fully take advantage of async
, you shouldn't return IEnumerable
, because it's synchronous (i.e. it blocks if no items are available). What you need is some sort of asynchronous collection, and you can use ISourceBlock
(specifically, TransformBlock
) from TPL Dataflow for that:
ISourceBlock<TDst> Foo<TSrc, TDest>(IEnumerable<TSrc> source)
{
var block = new TransformBlock<TSrc, TDest>(
async s => await ExecuteOrDownloadSomethingAsync(s),
new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded
});
foreach (var item in source)
block.Post(item);
block.Complete();
return block;
}
If the source is “slow” (i.e. you want to start processing the results from Foo()
before iterating source
is completed), you might want to move the foreach
and Complete()
call to a separate Task
. Even better solution would be to make source
into a ISourceBlock<TSrc>
too.