How does PLINQ Order input

Question

If I have the following order in an IEnumerable (list) :

1 - 2 - 3 - 4 - 5

and If I run a PLINQ query on this say :

list.AsParallel().AsOrdered().WithDegreeOfParallelism(10).Select(
                s => SomeDelegateFunction(s)).AsSequential().ToList();

For the above query I see in my logging (inside delegate function) that it uses multiple threads, but does not maintain the sequence of processing of the list. However for the below query i does maintain the sequence but uses a single thread to do the entire operation:

list.AsParallel().AsOrdered().WithDegreeOfParallelism(10).AsSequential().Select(
               s => SomeDelegateFunction(s)).ToList();

The difference between the two queries is the "AsSequential()" in the second query, the question I had is that when I use AsSequential() :

1 - Why does it not use multi threads? It could have broken down the work as :

1 - 2 (Give it to thread 1)
3 - 4 - 5 (Give it to thread 2)

Instead it does do 1 - 2 - 3 - 4 - 5 (In that order) but does it on a single thread - why??

Basically I need to process my list in the ORDER it came into as input but on multiple threads.

Any ideas ?

Answer 1

Basically I need to process my list in the ORDER it came into as input but on multiple threads.

Those are mutually exclusive requirements. You can do one or the other, but never both.

If you don't care what order the items were processed in, you just want to make sure that your end result is that the objects are in the order that they first came in as you can do something like this:

var list = new List<int> { 1, 2, 3, 4, 5 };

Parallel.For(0, list.Count, i =>
{
    list[i] = Process(i);
});

If you prefer PLINQ over Parallel.For you could do something like this:

Enumerable.Range(0, list.Count)
    .AsParallel()
    .WithDegreeOfParallelism(10)
    .ForAll(i =>
    {
        list[i] = Process(i);
    });

Answer 2

Answer to your first question

Why does it not use multi threads?

is that what AsSequential() does is exactly opposite to what AsParallel() does. AsParallel() makes a ParallelQuery<T> out on an ordinal enumerable IEnumerable<T> to allow parallel execution in multiple threads. AsSequential() is used to merge results of parallel execution into one enumerable back into the caller thread.

Answer to your second question

It could have broken down the work as :

1 - 2 (Give it to thread 1)
3 - 4 - 5 (Give it to thread 2)

Yes it could, but in case of parallel execution of this sequences 3 will probably process earlier then 1 and it will not be a sequential execution.

Nevertheless you can explicitly control how PLINQ breaks down the sequence into multiple partitions. So you can control yourself how your data gets distributed between multiple threads see Custom Partitioners for PLINQ

Answer 3

A multithreaded producer-consumer pattern would allow you to ensure that the items are processed in the same order, while allowing items to be processed concurrently. If you need to ensure that each item's processing completes before the next item's processing starts, however, you're out of luck, as Servy says.

In the producer-consumer pattern, to maintain order, you can use a thread-safe queue. One or more producer threads enqueues items to be processed; in this case, you would have one producer passing the items from the list, in order, to the queue. Multiple consumer threads can then dequeue the items, which will be in order.

See http://en.wikipedia.org/wiki/Producer-consumer_problem for more information.