How big can a memory-mapped file be?

Question

What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?

Answer 1

You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [2⁴⁰ to 2⁴⁰ + 1024] with two separate views.)

In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.

Answer 2

This has been my experience when using memory-mapped files under Win32:

If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.

If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.

I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.

Answer 3

There should be no other limits. Aren't those enough? ;-)

Answer 4

Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "

From MDSN.

I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.

Answer 5

With FUSE on linux you could also make an in-memory filesystem that extends to disk on demand. I'm not sure that qualifies as memory mapped, and the distinction gets kind of blurred.

Answer 6

Yes, there are limits to memory-mapped files. Most shockingly is:

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.

Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:

Overflow Error: memory mapped size must be positive

Big datasets are really a pain to work with.

Answer 7

Wikipedia entry on the subject: http://en.wikipedia.org/wiki/Memory-mapped_file