How dense are the 1s in the results? If they're sparse (i.e. mostly 0), find()
could shrink the logical masks down to numeric index lists. If you need further compression, depending on the size of X, you could convert the stored masks to uint32
or uint16
, which would take 1/2 or 1/4 the space of double
.
Then stash the results in a temporary variable that is still sliced by the loop index, and then combine them outside the parfor
loop, like you say.
masks = cell(1, X);
parfor i=1:X
data = output_of_some_expensive_function();
% do some other stuff with data here
% stash the loop results in a non-combining manner
masks{i} = uint32(find(data == 0));
end
% Combine in regular for loop
for i=1:X
isForeground(double(masks{i})) = 0;
end
If that's not enough, maybe you could write each result mask out to disk in a .mat file on a commonly-visible drive space or other persistent object store. If the workers for your parfor
are on the local machine, any drive will work, and should be fast because you may just go through your OS's disk cache.
You could also try transforming the output mask to a compressible form. E.g. run diff()
on it and zip the results in to a byte[]
"blob", and then do the inverse right before applying the mask in the regular for
loop. Depending on the structure of your masks, this may save additional space.
Note: logicals take more storage space than you might think; even though each element only has one bit of meaningful information, it takes a whole byte of memory, because that's the smallest addressable unit of memory.