I solved the problem using cellfun:
secs = cellfun(@(x) x(end-1:end),data{4}(:),'UniformOutput',false);
mins = cellfun(@(x) x(end-4:end-3),data{4}(:),'UniformOutput',false);
hours = cellfun(@(x) x(1:end-6),data{4}(:),'UniformOutput',false);
sec(:,1) = reshape(sscanf(sprintf('%s#', secs{:}), '%g#'), size(secs));
min(:,1) = reshape(sscanf(sprintf('%s#', mins{:}), '%g#'), size(mins));
hour(:,1) = reshape(sscanf(sprintf('%s#', hours{:}), '%g#'), size(hours));
EDIT:
This is a faster way that I found to do it. The cellfuns were taking too long, so I exploited MATLAB's lazy copying:
time = reshape(sscanf(sprintf('%s#',data{4}{1:totSamples}),'%d:%d:%d#'), 3,[])';
sec = time(:,3);
min = time(:,2);
hour = time(:,1);
clear time;