Speed increase for textscan use

Question

I did a program that takes a .csv file and stacks the 3rd column of each files in the appropriate 3rd dimension of a 512x512xNumberOfFiles cell array. The code goes like this :

[filenames,filepath] = uigetfile('*.csv','Opening the data files','','Multiselect','on');
filenames = fullfile(filepath,filenames);
NumFiles = numel(filenames);

Pixel = cell(512,512,NumFiles);

count=0;
num_pixels = size(Pixel,1)*size(Pixel,2);
for k = 1:NumFiles
    fid = fopen(char(filenames(k)));
    C = textscan(fid, '%d, %d, %d','HeaderLines',1);
    Pixel(count + sub2ind(size(Pixel),C{1}+1,C{2}+1)) = num2cell(C{3});
    count = count + num_pixels;
    fclose(fid);
end

The textscan call here takes approximately 0.5 +/- 0.03s per file I open (which is 262144 (512x512) data long), and my sub2ind call takes approximately 0.2 +/- 0.01s per file.

Is there any way to decrease this time or this seems like the most optimal way to run the code? I'll be working with approximately 1000 files each time, so waiting 8-9 minutes only to get the data right seems a bit excessive (considering I haven't used it yet for anything else).

Any tips?

Marc-Olivier

Answer 1

Hoping this would result in some improvement by still keeping it with textscan. Also, make sure the values look good.

Code

[filenames,filepath] = uigetfile('*.csv','Opening the data files',...
    '','Multiselect','on');
filenames = fullfile(filepath,filenames);
NumFiles = numel(filenames);

PixelDouble = NaN(512*512,NumFiles);
for k = 1:NumFiles
    fid = fopen(char(filenames(k)));
    C = textscan(fid, '%d, %d, %d','HeaderLines',1);
    PixelDouble(:,k) = C{3};
    fclose(fid);
end
Pixel = num2cell(permute(reshape(PixelDouble,512,512,[]),[2 1 3]))

I must encourage you to follow this question - Fastest Matlab file reading? and it's answers.