Increment ID in ParallelQuery.ForAll Method

Question

I have a list of files that needs to be processed at the same time. I tried to use ForAll extension method of the ParallelQuery class. I don't the files to be processed in order so I used ForAll.

Here's my sample code:

List<FileInfo> files = GetFilesToProcess();

files.AsParallel().ForAll(f => { // Process file here });

It works great but now I need to generate a unique integer ID for each of the file and I not sure how to do it without changing the AsParallel.ForAll to ForEach.

I read somewhere that I need to as an Interlocked but there will still be an issue.

Hope you can give me an idea here.

Thanks!

Answer 1

You can use Interlocked.Increment to generate the ID, or you can just use the index directly:

List<FileInfo> files = GetFilesToProcess();

files.AsParallel().Select((f, i) => new {File=f, ID=i})
     .ForAll(fp => 
           {
              FileInfo file = fp.File;
              int id = fp.ID; // ID is the index in the list

              // Process file here
           });

If you want to use Interlocked.Increment, you could do:

List<FileInfo> files = GetFilesToProcess();
int globalId = -1;

files.AsParallel().ForAll(f => 
                         { 
                              // Process file here 
                              int id = Interlocked.Increment(ref globalId);
                              // use ID
                         });

That being said, if your entire goal is to do "work" on a collection, I would recommend writing this as a Parallel.For or Parallel.ForEach instead. This is much more clear, as you're not using LINQ style syntax for the sole purpose of generating side effects:

List<FileInfo> files = GetFilesToProcess();
Parallel.For(0, files.Count, i =>
{
    var file = files[i];
    // Use i and file as needed
});

Answer 2

If you really must, then you could have an int you deal with through Interlocked.Increment.

Using the index of the source is a better bet, since that is information already available from the partitioner, and even such light sharing as Interlocked gives, is still sharing.

Alternatively, you could skip that and use some sort of UUID. In this case that's probably to heavy to be worth it (getting the index is nice and light). I mention it because the question "can I do this with zero sharing between the parallel tasks?" should always be considered, even if to be later dismissed.