How to allocate 16GB of memory in Go?

Question

I'm using the following simple Go code to allocate a 3D array of size 1024x1024x1024:

grid = make([][][]TColor, 1024)
for x = 0; x < 1024; x++ {
    grid[x] = make([][]TColor, 1024)
    for y = 0; y < 1024; y++ {
        grid[x][y] = make([]TColor, 1024)
    }
}

That TColor struct is a 4-component float64 vector:

type TColor struct { R, G, B, A float64 }

Halfway (x=477 and y=~600ish) through the allocation, the inner-most make() call panics with... runtime: out of memory: cannot allocate 65536-byte block (17179869184 in use)

This works fine with lower grid resolutions, ie 256³, 128³ etc. Now since the size of the struct is 4x4 bytes, that whole grid should require exactly 16 GB of memory. My machine (openSuse 12.1 64bit) has 32 GB of addressable physical (ie not-virtual) memory. Why can Go (weekly.2012-02-22) not allocate even half of this?

Answer 1

The struct has 4x8 bytes, not 4x4.

Answer 2

In the current implementation of the Go language, on 64-bit CPUs the Go runtime reserves 16GB of virtual memory from the operating system. This limits the total memory used by a Go program to 16GB.

If you plan to use Go in projects that require large amounts of memory you will need to edit the function runtime·mallocinit in file malloc.goc and increase the value of variable arena_size from 16GB to a bigger value (such as 32GB). After the edit, run

cd $GOROOT/src/pkg/runtime
go tool dist install -v

and then recompile your project.