Random access to large files (needing support 64-bit file offsets) in node.js?

Question

I'm thinking of porting some of my cross-platform scripts to node.js partly to learn node.js, partly because I'm more familiar with JavaScript these days, and partly due to problems with large file support in other scripting languages.

Some scripting languages seem to have patchy support for large file offsets, depending on such things as whether they are running on a 32-/64-bit OS or processor, or need to be specifically compiled with certain flags.

So I want to experiment with node.js anyway but Googling I'm not finding much either way on its support (or it's library/framework support etc) for large files with 64-bit offsets.

I realize that to some extend this will depend on JavaScript's underlying integer support at least. If I correctly read What is JavaScript's Max Int? What's the highest Integer value a Number can go to without losing precision? it seems that JavaScript uses floating point internally even for integers and therefore

the largest exact integral value is 2⁵³

Then again node.js is intended for servers and servers should expect large file support.

Does node.js support 64-bit file offsets?

---

UPDATE

Despite the _LARGEFILE_SOURCE and _FILE_OFFSET_BITS build flags, now that I've started porting my project that requires this, I've found that fs.read(files.d.fd, chunk, 0, 1023, 0x7fffffff, function (err, bytesRead, data) succeeds but 0x80000000 fails with EINVAL. This is with version v0.6.11 running on 32-bit Windows 7.

So far I'm not sure whether this is a limitation only in fs, a bug in node.js, or a problem only on Windows builds.

Is it intended that greater-than-31-bit file offsets work in node.js in all core modules on all platforms?

Answer 1

It was a little difficult to track down but node.js has only supported 64-bit file offsets since version 0.7.9 (unstable), from the end of May 2012. In stable versions from version 0.8.0, from the end of June 2012.

fs: 64bit offsets for fs calls (Igor Zinkovsky)

On earlier versions failure modes when using larger offsets, failure modes vary from silently seeking to the beginning of the file to throwing an exception with EINVAL.

See the (now closed) bug report:

File offsets over 31 bits are not supported

To check for large file support programatically from node.js code

if (process.version.substring(1).split('.') >= [0,7,9]) {
  // use 64-bit file offsets...
}

Answer 2

Node.js is compiled with _LARGEFILE_SOURCE and _FILE_OFFSET_BITS on all platforms, so internally it should be safe for large file access. (See the common.gypi in the root of the source dir.)

In terms of the libraries, it uses Number for start (and end) options when creating read and write streams (see fs.createReadStream). This means you can address up to position 2^53 through node (as evidenced here: Also relevant: What is JavaScript's highest integer value that a Number can go to without losing precision?) This is visible in the lib/fs.js code.