Question

I need to operate in big 3-dim non-sparse matrices in Matlab. Using pure vectorization gives a high computation time. So, I have tried to split the operations into 10 blocks and then parse the results. I got surprised when I saw the the pure vectorization does not scale very well with the data size as presented in the following figure.

enter image description here

I include an example of the two approaches.

% Parameters:
M = 1e6;  N = 50;  L = 4;  K = 10;

% Method 1: Pure vectorization
mat1 = randi(L,[M,N,L]);
mat2 = repmat(permute(1:L,[3 1 2]),M,N);
result1 = nnz(mat1>mat2)./(M+N+L);

% Method 2: Split computations
result2 = 0;
for ii=1:K
    mat1 = randi(L,[M/K,N,L]);
    mat2 = repmat(permute(1:L,[3 1 2]),M/K,N);
    result2 = result2 + nnz(mat1>mat2);
end
result2 = result2/(M+N+L);

Hence, I wonder if there is any other approach that makes big matrix operations in Matlab more efficient. I know it is a quite broad question, but I will take the risk :)


Edit:

Using the implementation of @Shai

% Method 3
mat3 = randi(L,[M,N,L]);
result3 = nnz(bsxfun( @gt, mat3, permute( 1:L, [3 1 2] ) ))./(M+N+L);

The times are:

enter image description here

Was it helpful?

Solution

Why repmat and not bsxfun?

result = nnz(bsxfun( @gt, mat1, permute( 1:L, [3 1 2] ) ))./(M+N+L);

It seems like you are using up your RAM and the OS starts to allocate room in swap for the very large matrics. Memory swapping is always a very time consuming operation and it gets worse as the amount of memory you require increases.
I believe you are witnessing thrashing.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top