NodeJS large directory file changes

Question

I am working on more of a security dashboard, it watches for changes in files in the entire home directory with hundreds of sites (all Joomla, so a lot of files).

In order to keep on top of potential security issues we want to watch for file changes in an efficient way without creating unnecessary CPU/Memory overhead. We want to watch it at a faster interval but I know its more of a balancing act when you do want to keep it from using more cpu then a side process should.

I have tried to use "watch" with the following code, running in the home directory:

var watch, fs;
watch = require('watch');
fs = require('fs');

watch.createMonitor(__dirname,{interval:500,filter:function(file,stat){
    if(file.indexOf('index.php')!=-1){
        return true;
    }else{
        return false;
    }
}},function(monitor){
    monitor.filter(function(file){
        console.log(file);
    })
    monitor.on('created',function(file,stat){
        console.log(file + ' new');
    });
    monitor.on('changed',function(file,stat){
        console.log(file + ' changed');
    });
    monitor.on('removed',function(file,stat){
        console.log(file + ' deleted');
    });
});

However this spikes the CPU to over 100% of a single core (sometimes 2) out of 8. Memory also takes up about 20% of 8gb pretty quickly as well. This is all just to create the watch event on all the files, so its before it can actually detect any file changes.

I know the issue with this is it goes through each file individually, and only does not track it if you filter that sort of file. Typically all I need to watch is the index.php in every directory, down to a point that it could be consistent (with some exceptions).

Is there a module already built to do this? Or is this something new? All modules I find assume its a smaller directory (like watching LESS or something) So not built for this sort of application at all.

Any ideas? I know this code will need to be scrapped as there is no way I can see to stop the CPU overhead.

Answer 1

Do not use package 'watch', just use fs.watch(...)

package 'watch':

• consistent APIs
• very slow because implement mostly in node, look source to see how it work
• souce code: https://github.com/mikeal/watch/blob/master/main.js

fs.watch(..)

• non-consistent APIs, not all OSs are supported.
• very fast because it reused OS features
• document: http://nodejs.org/docs/latest/api/fs.html#fs_fs_watch_filename_options_listener