As lines are added to a file, how do I detect and read them in using a single file descriptor or stream?

Question

Background -

I'm trying to use node.js and the fs module to accomplish an end goal of monitoring a file, and detecting the lines that have been appended to it.

Current Implementation -

I'm currently using fs.watch to monitor the changes to the file persistently, and fs.readFile to read the file once the watch has been triggered.

Drawbacks -

The downside of this implementation is that it is computationally expensive and slow to derive the appended lines in this manner, especially since it requires reading in the entire file contents despite my interest in only the appended lines.

Ideal Solution -

I would like to instead use fs.createReadStream to somehow read the file up until the end, leave the file descriptor at the end, and start reading again once the file has been appended to.

I've found two ways to read the contents of a stream buffer, but in both implementations, which are readable.read() and readable.on('data',...), it seems the stream is ended once there is no more data to read, although the stream is not closed. I'm not exactly sure how to continue using a ended stream, as readable.resume() does not seem to do anything.

My Question -

How do I read appended lines from a file in a way that is triggered once the file is modified? Is my ideal solution down the right track?

Thank you for your time and help!

Answer 1

This is a problem I once had, and it was quite a headache. This is the implementation that I came up with.

var fs = require('fs');

var file = __dirname + '/file.log';
fs.stat(file, function(err, stats) {
  var start = stats.size;
  // read the entire file here if you need it
  watchFile(start);
});

function watchFile(start) {
  fs.watch(file, function(event, filename) {
    fs.stat(file, function(err, stats) {
      var stream = fs.createReadStream(file, {
        start: start,
        end: stats.size
      });

      var lines = new String();
      stream.on('data', function(data) {
        lines += data;
      });

      stream.on('end', function() {
        // you have the new lines
      });

      start = stats.size + 1;
    });
  });
};

First I find the size of the file, and pass it to a watch function. Every time the file changes, I find out the new size of the file and read from the old position to the new position. On some systems the watch function might fire twice per change, so you might want to add checks to get rid of useless reads such as when the start and end are the same byte.