malloc sometimes fails. or pread sometimes fails

Question

so I have a test program which reads huge amounds of data into a buffer and mallocs the buffer accordingly. However it malloc fails on huge sizes.

Is there a way to fix this?

thanks for any replies

the device /dev/sdc is a 2TB disk.

Here is ready 2 compile code:

#define          _FILE_OFFSET_BITS                            64     
#define          BLKGETSIZE64                                _IOR(0x12,114,size_t)    
#define          _POSIX_C_SOURCE                             200809L

#include <stdio.h>    
#include <inttypes.h>    
#include <stdlib.h>    
#include <fcntl.h>    
#include <unistd.h>

int readdata(int fp,uint64_t seekpoint, uint64_t seekwidth) {

    int16_t  *buf;

    buf=(int16_t *)malloc(seekwidth*sizeof(int16_t)+1);
    if (buf==0) {
      printf("ERROR malloc(%"PRIu64")\n",seekwidth*(sizeof(int16_t)));
      return 3;
    }

    if (pread(fp,buf,seekwidth,seekpoint)==seekwidth) {
        printf("SUCCES READING AT: %"PRIu64"| WITH READ WIDTH: %"PRIu64"\n",seekpoint,seekwidth);
        free(buf);
        return 1;
    } else {
        printf("ERROR READING AT: %"PRIu64"| WITH READ WIDTH: %"PRIu64"\n",seekpoint,seekwidth);
        free(buf);
        return 2;
    }

}



int main() {
    uint64_t    readwith,
                offset;
    int         fp=open("/dev/sdc",O_RDWR);

    readwith=10000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=100000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=1000000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=10000000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=100000000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=1000000000;   offset=0;
    readdata(fp,offset,readwith);
    readwith=10000000000;   offset=0;
    readdata(fp,offset,readwith);
    close(fp);

}

output on my system is :

SUCCES READING AT: 0| WITH READ WIDTH: 10000
SUCCES READING AT: 0| WITH READ WIDTH: 100000
SUCCES READING AT: 0| WITH READ WIDTH: 1000000
SUCCES READING AT: 0| WITH READ WIDTH: 10000000
SUCCES READING AT: 0| WITH READ WIDTH: 100000000
SUCCES READING AT: 0| WITH READ WIDTH: 1000000000
ERROR READING AT: 0| WITH READ WIDTH: 10000000000
ERROR READING AT: 0| WITH READ WIDTH: 100000000000
ERROR READING AT: 0| WITH READ WIDTH: 1000000000000
ERROR READING AT: 0| WITH READ WIDTH: 10000000000000
ERROR READING AT: 0| WITH READ WIDTH: 100000000000000
ERROR READING AT: 0| WITH READ WIDTH: 1000000000000000

and sometimes

ERROR malloc(2000000000)
ERROR READING AT: 0| WITH READ WIDTH: 10000000000

Answer 1

Well, i would say you can't allocate the memory of size 10000000000, because that is 9536 MiB, and I guess you don't have that much RAM.

You may consider using STXXL.

The core of STXXL is an implementation of the C++ standard template library STL for external memory (out-of-core) computations, i. e., STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks.

Answer 2

The Doug lea malloc(dlmalloc) about the minimum and maximum size of chunk

If n is zero, malloc returns a minimum-sized chunk. (The minimum size is 16 bytes on most 32bit systems, and 32 bytes on 64bit systems.) Note that size_t is an unsigned type, so calls with arguments that would be negative if signed are interpreted as requests for huge amounts of space, which will often fail. The maximum supported value of n differs across systems, **but is in all cases less than the maximum representable value of a size_t.

The maximum chunk size could be(2^31) and its around 2147483648.

So in your case the last successful malloc call is for 1000000000*2 is larger than above maximum value hence malloc returns the NULL.

Answer 3

If you want to do something with the data on a whole disk or huge file, you probably want to use mmap(2) instead.

From the man page:

mmap() creates a new mapping in the virtual address space of the calling process.

int fd = open("/dev/sdc", O_RDONLY);
unsigned char *data = mmap(NULL, length, PROT_READ, MAP_SHARED, fd, 0);

Things get more tricky and dangerous if you want to be able to write to the disk, but this is also possible (see the mmap(2) man page for more details.)