nodejs: atomic file replace operation, trigger only one time some watchers

Question

I have two kinds of processes. One shall write a file, the others shall read it, if it is changed.

I found fs.watch to get the reading processes informed, if they have to reload the file.

To write the file, I could use fs.writeFile.

But I am not sure, if this is a atomic replace. Will watch trigged several times, if the file is large?

What would be the correct way, to get this as atomic operation?

FYI: On the target system runs Linux.

Answer 1

After some test I got this results:

fs.watch fires the change event multiple times while large file is written. The problem is, you can not see, which will be the last one to figure out, that the write operation has finished.

The solution is: Write a temp file in the same directory and than use link and unlink to replace the old file with the new one.

In code it could look like this:

var fs = require("fs");
var path = require("path");
var consts = require('constants');

/**
 * write given date in temp file and replace the dstpath with that.
 *
 * @author Tobias Lindig
 * 
 * @param  {string} dstpath destination path
 * @param  {string} data    [description]
 *
 * @throws {Error} If fs operations failed.
 */
function atomicWrite(dstpath, data) {
    // build a unique temp path
    var dir = path.dirname(dstpath);
    var tempName = [
        'temp',
        process.pid,
        (Math.random() * 0x1000000000).toString(36),
        new Date().getTime()
    ].join('-');
    var tempPath = path.join(dir, tempName);

    var tempOptions = {
        encoding: 'utf8',
        mode: 420, // aka  0644 in Octal
        flags: consts.O_CREAT | consts.O_TRUNC | consts.O_RDWR | consts.O_EXCL
    };

    //local function to remove temp file
    var fn_removeTempFileSync = function() {
        try {
            fs.unlinkSync(tempPath);
        } catch (ex) {
            // ignore fail, may be file was not created.
        }
    };

    try {
        fs.writeFileSync(tempPath, data, tempOptions);

        // remove possible old version of file
        // because fs.link can not overwrite existing dstpath.
        if (fs.existsSync(dstpath)) {
            fs.unlinkSync(dstpath);
        }

        //copy temp file to destination path
        fs.linkSync(tempPath, dstpath);

        fn_removeTempFileSync();
    } catch (ex) {
        fn_removeTempFileSync();
        throw ex;
    }
}

Now fs.watch fires the event change for the file ("dstpath") only one time.

That the theory.

But in the real world, unfortunately the change event was not triggered in every case, some time it missed. So I have watch the rename event too.

The event will come in this order:

rename // ever, if file was deleted rename // ever, file was created change // some times, file was created

To read the file only one time, I member the mtime of file and read only, if it was different.