How to reliable observe a directory for newly created files?

Question

I've to observe a specific directory for new files being created with a specific file ending (for example: .txt).

This has to work regardless of the underlying operating system (Windows/Linux) or if it's an NFS share being accessed or a Samba share or a local directory.

At first I only used FileSystemWatcher

 watcherForPathToWatch = new FileSystemWatcher(folderToWatch, "*.rdy");
 watcherForPathToWatch.Created += WatcherForPathToWatch_Created;
 watcherForPathToWatch.Error += WatcherForPathToWatch_Error;
 watcherForPathToWatch.IncludeSubdirectories = false;
 watcherForPathToWatch.EnableRaisingEvents = true;

Content of WatcherForPathToWatch_Created:

ThreadPool.QueueUserWorkItem(new WaitCallback(delegate { HandleFile(e.FullPath); }));

The problem is that it doesn't pick up all files. Even on a local directory on a NTFS partition there are sometimes files not being handled.

So, without thinking too much about it, I thought doing a DirectoryEnumeration every few minutes would solve my problem. Looks like this

        private void EnumerationBasedArchiving()
        {
            watcherForPathToWatch.EnableRaisingEvents = false;                
            int maxThreads;
            int unneeded;
            int currThreads;
            do
            {
                Thread.Sleep(500);
                ThreadPool.GetMaxThreads(out maxThreads, out unneeded);
                ThreadPool.GetAvailableThreads(out currThreads, out unneeded);
            } while ((maxThreads - currThreads) != 0);
            List<string> listOfFilesToArchive = null;
            do
            {
                try
                {
                    Directory.EnumerateFiles(folderToWatch, "*.txt", SearchOption.TopDirectoryOnly).ToList();
                }
                catch (Exception ex)
                {
                    //error logging happens here, too
                    Thread.Sleep(10000);
                }
            } while (listOfFilesToArchive == null);
            watcherForPathToWatch.EnableRaisingEvents = true;
            Parallel.ForEach(listOfFilesToArchive, (fileToArchive) => { HandleFiles(fileToArchive); });
        }

The problem I've with this approach is, while it should work in most situations, it's not too reliable. Especially the point between enumerating the directory's content and switching the handler back on. If there's a file created exactly between those two points, I won't handle that until the next enumeration cycle.

Now I wonder if the 'best' solution would be to completely disregard the FileSystemWatcher or if there's a way to somehow make this more reliable, since it's kind of important to handle those files in the order they were created.

Answer 1

There is inotify on Linux described here, but you will not get anything that covers Windows and Linux and NFS and Samba, so you are probably going to end up rolling your own.

Just a thought.. you said that it's important to handle the files in the order they are created... well just because you may find the files in the "wrong" order that doesn't necessarily prevent you from processing them in the correct order - because each file has a timestamp so you can process them in a different order from the order you find them. Of course, that may involve coding a delay between when you find them and when you actually process them, but if your polling interval is set right...