Can I add to, and yield from a ConcurrentQueue in parallel?

Question

I have a class that goes to get objects from a web service. My consumption of these objects is not order dependent so I issue my web requests in parallel and am adding the results to a ConcurrentQueue.

At the same time, as the requests are being made, we can process the responses.

So I would like to be able to iterate over the contents of the ConcurrentQueue while my items are being added to it.

How can I have one class fill up the queue and the other class empty it. It will almost certainly be emptied faster than it is filler, so I can't just enumerate it without having a yield because the collection will be empty and won't wait.

Answer 1

Use a BlockingCollection to wrap the ConcurrentQueue (the default backing store for a BlockingCollection is a ConcurrentQueue, so you don't even need to be explicit about it.

This allows you to write the consumer as:

foreach(var item in queue.GetConsumingEnumerable())
    Process(item);

Answer 2

1. You can use a BlockingCollection.

2. You can use an AutoResetEvent; the consumer waits on it whenever it hits on empty, while the producer(s) set it after adds (or batches of adds if that suits the pattern better).

3. You can have the producer threads do the actual processing.

The first would be the most straight-forward. The second is pretty much a way to implement the first, so doesn't gain you much, but can be useful if you ever have a need to change things from the normal pattern for some reason. The third is much messier, but I mention it because if your final output combines results in a complicated way it can sometimes turn out to be convenient.