What would be a good design to achieve this synchronization across machines..?

Question

I have 2 machines both running one process each. shell process on Machine A will scp something to machine B, and java process on B will use these files. Both processes run as crontab tasks.

How to achieve synchronization/atomicity etc? How to signal that whole of file has been written.

so that process on B always has access to latest and complete files, the handler doesn't go stale..

Answer 1

Assuming you're using a filesystem with atomic moves you can do that. Or use symbolic links.

A copies the file to a temp location on B. When the upload is complete, it relocates, with a move or symlink, the file into the expected location. B can then only ever see completely uploaded files.

If your process on A cannot SSH into B to make the final move, it could rather add another, zero byte marker file which indicates the upload is complete.

A uploads FOO.txt, when that upload is complete, it creates the FOO.txt.done file. B then scans the directory for *.done files, and uses the associated data file. Plus cleanup of course.

Answer 2

A quick'n'dirty work around / solution for the problem could be resolved by using user control, setting the ownership (and chmod'ing appropriately) when copying the file (using SCP from A), when finishing then changing the ownership / permissions so that the process on B will be able to access the file.

Answer 3

A few hacks that I can suggest I guess. You can either :-

1. create a temp file (like a lock file) at the start of file writing, the presence of which means that the file being streamed is being written to and delete the temp file once the writing is complete. OR
2. Remove crontab on the consumer process and have the scp process send a signal (JMS/ unicast). The java process could just be a listener on a queue/ socket or be invoked upon receiving a unicast.