Nodejs process locks up reading from blocking FUSE file

Question

I have a nodejs webserver which reads files and serves the content. Some of the files are FUSE virtual files which may block reads for long periods of time while their backing services wait for data to provide.

It seems that if 5 of these read requests pile up, the process will no longer read from any files. Also, I'm not sure how to terminate these pending reads if necessary.

Is this a limitation of nodejs or is the FUSE filesystem behaving in a manner inconsistent with FUSE's expectations?

Answer 1

According to this NodeJS issue report #7256, if a thread in the underlying libuv I/O thread pool attempts to read from a file that blocks, the thread will hang until the device or process backing the target file yields control to the thread, ie the read call returns.

Furthermore, NodeJS v0.10 on *nix systems relies on a fixed size thread pool with a default size of 4, according to this libuv issue report #649. This explains why the system appeared to lock-up after ~5 read attempts.

The situation can be worked around in several ways:

1. Increase the size of underlying libuv thread pool using the UV_THREADPOOL_SIZE environment variable. This glosses over the issue by simply allowing more opportunities to read before the Node process begins to suffer.

2. Open the blocking files using the recently exported O_NONBLOCK flag. This change is a part of the Node 0.10 code stream and I have confirmed that it works as expected (having compiled Node from source with _XOPEN_SOURCE set) but is not in the latest 0.10.28 release at the time of this comment. It seems reasonable to believe that the nonblock functionality will be a part of the 0.10.29 release. The device backing the file must also respect the O_NONBLOCK flag for this to work properly.

Some code to demonstrate option #2

var constants = process.binding('constants');
fs.open('/path/to/file', constants.O_NONBLOCK, function(err,fd){
    fs.read(fd, buf, 0, 10, null, function(err, bytesRead, buffer){
        // Read should return immediately
        if(err.code === 'EAGAIN'){
            // Not ready to read, try again later
        } else {
            // Do something
        }
    });
});

Answer 2

are you using readFileSync to read from these files? If so, thats your problem right there.

Node should be able to wait for a large number of resources while still serving requests. Show us your code to read from the files.