Prevent multiple threads writing the same file

Question

i am designing and developing an api where multiple threads are downloading files from the net and then write it to disk.

if it is used incorrectly it could happen that the same file is downloaded and written by more than one threads, which will lead to an exception at the moment of writing to disk.

i would like to avoid this problem with a lock() { ... } around the part that writes the file, but obviously i dont want to lock with a global object, just something that is related to that specific file so that not all threads are locked when a file is written.

i hope this question is understandable.

Answer 1

You may consider using ReaderWriterLockSlim

http://msdn.microsoft.com/en-us/library/system.threading.readerwriterlockslim.aspx

private ReaderWriterLockSlim fileLock = new ReaderWriterLockSlim();


fileLock.EnterWriteLock();
try
{
   //write your file here
}
finally
{
    fileLock.ExitWriteLock();
}

Answer 2

So what you want to be able to do is synchronize a bunch of actions based no a given key. In this case, that key can be an absolute file name. We can implement this as a dictionary that maps a key to some synchronization object. This could be either an object to lock on, if we want to implement a blocking synchronization mechanism, or a Task if we want to represent an asynchronous method of running the code when appropriate; I went with the later. I also went with a ConcurrentDictionary to let it handle the synchronization, rather than handling it manually, and used Lazy to ensure that each task was created exactly once:

public class KeyedSynchronizer<TKey>
{
    private ConcurrentDictionary<TKey, Lazy<Task>> dictionary;
    public KeyedSynchronizer(IEqualityComparer<TKey> comparer = null)
    {
        dictionary = new ConcurrentDictionary<TKey, Lazy<Task>>(
            comparer ?? EqualityComparer<TKey>.Default);
    }

    public Task ActOnKey(TKey key, Action action)
    {
        var dictionaryValue = dictionary.AddOrUpdate(key,
            new Lazy<Task>(() => Task.Run(action)),
                (_, task) => new Lazy<Task>(() =>
                    task.Value.ContinueWith(t => action())));
        return dictionaryValue.Value;
    }

    public static readonly KeyedSynchronizer<TKey> Default =
        new KeyedSynchronizer<TKey>();
}

You can now create an instance of this synchronizer, and then specify actions along with the keys (files) that they correspond to. You can be confident that the actions won't be executed until any previous actions on that file have completed. If you want to wait until that action completes, then you can Wait on the task, if you don't have any need to wait, then you can just not. This also allows you to do your processing asynchronously by awaiting the task.

Answer 3

I had a similar situation, and resolved it by lock()ing on the StreamWriter object in question:

private Dictionary<string, StreamWriter>  _writers;  // Consider using a thread-safe dictionary

void  WriteContent(string file, string content)
    {
    StreamWriter  writer;
    if (_writers.TryGetValue(file, out writer))
        lock (writer)
            writer.Write(content);
    // Else handle missing writer
    }

That's from memory, it may not compile. I'd read up on Andrew's solution (I will be), as it may be more exactly what you need... but this is super-simple, if you just want a quick-and-dirty.

Answer 4

I'll make it an answer with some explanation.

Windows already have something like you want, idea behind is simple: to allow multiple processes access same file and to carry on all writing/reading operations, so that: 1) all processes operates with the most recent data of that file 2) multiple writing or reading occurs without waiting (if possible).

It's called Memory-Mapped Files. I was using it for IPC mostly (without file), so can't provide an example, but there should be some.

You could mimic MMF behavior by using some buffer and sort of layer on top of it, which will redirect all reading/writing operations to that buffer and periodically flush updated content into physical file.

P.S: try to look also for file-sharing (open file for shared reading/writing).