Matlab: your opinion about a small memory issue working with matrix

Question

I have a small question regarding MATLAB memory consumption.

My Architecture:

- Linux OpenSuse 12.3 64bit
- 16 GB of RAM
- Matlab 2013a 64 bit

I handle a matrix of double with size: 62 x 11969100 (called y)

When I try the following:

a = bsxfun(@minus,y,-1)

or simply

a = minus(y, -1)

I got a OUT of MEMORY error (in both cases).

I've just computed the ram space allocated for the matrix:
62 x 11969100 x 8 = 5.53 GB

Where am I wrong?!

Thanks a lot!

Answer 1

I'm running on Win64, with 16GB RAM.

Starting with a fresh MATLAB, with only a couple of other inconsequential applications open, my baseline memory usage is about 3.8GB. When I create y, that increases to 9.3GB (9.3-3.8 = 5.5GB, about what you calculate). When I then run a = minus(y, -1), I don't run out of memory, but it goes up to about 14.4GB.

You wouldn't need much extra memory to have been taken away (1.6GB at most) for that to cause an out of memory error.

In addition, when MATLAB stores an array, it requires a contiguous block of memory to do so. If your memory was a little fragmented - perhaps you had a couple of other tiny variables that happened to be stored right in the middle of one of those 5.5GB blocks - you would also get an out of memory error (you can sometimes avoid that issue with the use of pack).

Answer 2

The output of memory on windows platform:

>> memory
Maximum possible array:            2046 MB (2.145e+009 bytes) *
Memory available for all arrays:   3226 MB (3.382e+009 bytes) **
Memory used by MATLAB:              598 MB (6.272e+008 bytes)
Physical Memory (RAM):             3561 MB (3.734e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

The output of computer on linux/mac:

>> [~,maxSize] = computer
maxSize =
     2.814749767106550e+14    % Max. number of elements in a single array

with some hacks (found here):

>> java.lang.Runtime.getRuntime.maxMemory
ans =
    188416000  

>> java.lang.Runtime.getRuntime.totalMemory
ans =
    65011712

>> java.lang.Runtime.getRuntime.freeMemory
ans =
    57532968

As you can see, aside from memory limitations per variable, there are also limitations on total storage for all variables. This is not different for Windows or Linux.

The important thing to note is that for example on my Windows machine, it is impossible to create two 1.7GB variables, even though I have enough RAM, and neither is limited by maximum variable size.

Since carrying out the minus operation will assign a result of equal size to a new variable (a in your case, or ans when not assigning to anything), there need to be at least two of these humongous things in memory.

My guess is you run into the second limit of total memory space available for all variables.

Answer 3

bsxfun is vectorized for efficiency. Typically vectorized solutions require more than just minimal memory.

You could try using repmat, or if that does not work a simple for loop.

In general I believe the for loop will require the least memory.