How to sync 2 or more watched folders

Question

We need to implement a feature to our program that would sync 2 or more watched folders.

In reality, the folders will reside on different computers on the local network, but to narrow down the problem, let's assume the tool runs on a single computer, and has a list of watched folders that it needs to sync, so any changes to one folder should propagate to all others.

There are several problems I've thought about so far:

• Deleting files is a valid change, so if folder A has a file but folder B doesn't, it could mean that the file was created in folder A and needs to propagate to folder B, but it could also mean that the file was deleted in folder B and needs to propagate to folder A.

• Files might be changed/deleted simultaneously in several directories, and with conflicting changes, I need to somehow resolve the conflicts.

• One or more of the folders might be offline at any time, so changes must be stored and later propagated to it when it comes online.

I am not sure what kind of help if any the community can offer here, but I'm thinking about these:

• If you know of a tool that already does this, please point it out. Our product is closed-source and commercial, however, so its license must be compatible with that for us to be able to use it.

• If you know of any existing literature or research on the problem (papers and such), please link to it. I assume that this problem would have been researched already.

• Or if you have general advice on the best way to approach this problem, which algorithms to use, how to solve conflicts, or race conditions if they exists, and other gotchas.

The OS is Windows, and I will be using Qt and C++ to implement it, if no tools or libraries exist.

Answer 1

It's not exceptionally hard. You just need to compare the relevant change journal records. Of course, in a distributed network you have to assume the clocks are synchronized.

And yes, if a complex file (anything you can't parse) is edited while the network is split, you cannot avoid problems. This is known as the CAP theorem . Your system cannot be Consistent, Always Available and also resistant against Partitioning (going offline)