mmap Bus Error writing to MAP_SHARED file over 2Gb

Question

I am trying to create a memory mapped file using MAP_SHARED. I run into issues when the file size reaches 2gb. The code pasted below is what I am using (as a test).

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>

#define MY_FILE "/dev/shm/mmap_test"
#define BLOCK_SIZE (1024*1024)
#define NUM_FILES 1

void mk_file(int f_num)
{ 
    uint64_t len = 0;
    int fd, j, k;
    char tmp_file[1024], *x, *rr_addr;

    // Open file in /dev/shm
    sprintf(tmp_file, "%s%d", MY_FILE, f_num);
    fd = open(tmp_file, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);

    if (fd == -1)
    {
        perror("file open");
        exit(-1);
    }

    // 16Gb file 
    len = 16UL * 1024 * 1024 * 1024;
    printf("len: %ld Gb\n", len/(1024*1024*1024));

    printf("Mapping %ld blocks\n", len/BLOCK_SIZE);

    for (j = 0; j < len/BLOCK_SIZE; j++) {
        // Increase the file size
        ftruncate(fd, ((j + 1) * BLOCK_SIZE));

        // Just mmap memory... don't have file backing
        //rr_addr = mmap(NULL, BLOCK_SIZE, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE|MAP_NORESERVE, -1, 0);

        // MMAP a region to the file at a given offset
        rr_addr = mmap(NULL, BLOCK_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_NORESERVE, fd, (j * BLOCK_SIZE));

        if (rr_addr == MAP_FAILED) {
            perror("mmap error");
            printf("INDEX: %d\n", j);
            exit(-1);
        }

        // Write to every byte of allocated memory
        x = (char *) rr_addr;

        for (k = 0; k < BLOCK_SIZE; k++)
        {
            *x = '1';
            x++;
        }
    }

    return;
}

int main(int argc, char **argv)
{
    uint64_t i;

    for (i = 0; i < NUM_FILES; i++)
       mk_file(i);

    return 0;
}

In the code above, I get a bus error when the offset in the file reaches 2gb. These are the things I have tried:

1. If I change NUM_FILES to 16 and len to 1GB, I don't have any issues.
2. If I remove the for loop which writes to the memory (only mmap), the program does not crash (even for len much greater than 2gb) b/c the linux kernel does not actually map pages to the file until you read/write to the mmap'ed region.
3. If I change the mmap call from MAP_SHARED to MAP_ANON (uncomment the first mmap call and comment out the second one) and not linked to a file, there are no issues (even the writes succeed).
4. There is enough space on /dev/shm (30gb), I am using only 16gb here.
5. I don't have to write to every allocated byte. I only need to write to the last mmap'ed region (move the inner for loop outside) and if the offset + BLOCK_SIZE >= 2gb then I get a bus error.
6. I have tried this on Ubuntu 13.10 and CentOS 6.4 and both have the same issue.

I am wondering if this is an issue in the linux kernel? Has anyone tried mmap'ing a single file using MAP_SHARED greater than 2gb and used (read/write to) it successfully?

Answer 1

I believe the problem is that j is an int. When j hits large values, (j + 1) * BLOCK_SIZE overflows and your ftruncate call does not do what you intend. Checking the return value from ftruncate should confirm this.

The mmap man page specifically calls out SIGBUS as meaning that the attempted access is not backed by the file.

Answer 2

I am not sure if this helps in your case. When we had problems with big files the following helped. We have put the macro:

#define _FILE_OFFSET_BITS 64

before including standard headers. You can define it also on command line when invoking gcc.