How can I sequentially process elements from a queue in C# .NET

Question

I am writing software for Windows in Visual Studio 2012, coded in C#.

By using the FileSystemWatcher class, I am monitoring a directory for changes: if one or more new files are added to the directory, an event gets fired by FileSystemEventHandler class.

This event adds the file name(s) to an array.

I then want to process the array, that is: process every file (access via the file name) in the array and do something with the file.

Now: I need to make sure the file processing happens sequentially. Look at file 1, process it, and only when this is finished, look at file 2, process it, and so on.

The file processing takes about 5 seconds per file. Meanwhile, new files can be added to the directory and thus the FileSystemEventHandler class will fire because of these changes. But I do not want this to happen - there should always only be one file at a time that gets processed - regardless of how many times the event for changes gets fired!

Can anyone provide me with a pattern (or code snippet) for this, please?!

Answer 1

I think following class would be optimal,

BlockingCollection<T> Class

Provides blocking and bounding capabilities for thread-safe collections that implement IProducerConsumerCollection.

It acts as concurrent queue and you can process items async using Tasks.

Alternative would be ConcurrentQueue<T>, But The BlockingCollection gives you two important features

It's thread-safe

When you call Take(), it will block(i.e. wait until something is in the queue) for you, so you don't have to write any code with ManualResetEvents and the like, which is a nice simplification.

Answer 2

You can also choose to use the RX framework.

Something like this:

public static Tuple<FileSystemWatcher, Task> Watch(this string path, Action<string> processNewFile)
{
    var watcher = new FileSystemWatcher(path) {EnableRaisingEvents = true};
    return
        Tuple.Create(
            watcher,
            Observable
                .FromEventPattern<FileSystemEventHandler, FileSystemEventArgs>(
                    handler => watcher.Created += handler,
                    handler => watcher.Created -= handler)
                .Select(pattern => pattern.EventArgs.FullPath)
                .ForEachAsync(processNewFile));
}

This approach is async-friendly, concise and removes the need to write a lot of boilerplate code.

Answer 3

The pattern you want is one where the FileSystemEventHandler simply adds file names to a shared queue.

Then this queue can be watched independently by another thread, which then removes items from the queue and processes them.

This way, there's no need to 'stop' the event handler from adding items whilst you're processing a file. It can keep adding more, and it won't affect the worker thread.

I would suggest something like the ConcurrentQueue class to manage your list of files to be processed.

The following sample code by Microsoft shows how to use a BlockingCollection (which wraps ConcurrentQueue) to achieve this:

http://msdn.microsoft.com/en-us/library/dd997371.aspx

Answer 4

You need to use Queue . the queue needs to be handled in a separate thread. so you can handle the files array list without interfering with the watch folder event

Answer 5

You can set a boolean variable, which is set to true if another file is processing and false after its done processing. Meanwhile if the FileSystemEventHandler fires a new event, it checks the variable, which, if is set to true, will wait. Otherwise, set the variable to true itself and carryon.