I would sort by the 5th row without using sortrows
. Then you only have to convert the 5th row to double:
%sorted indices
[~,ix]=sort(str2double(M(:,5)));
%get mask of nan-rows to be ignored
nanmask=isnan(str2double(M(ix,5)))
%get non-nan indices in reverse order
ix=flipdim(ix(~nanmask),1)
%sort
sortedM=M(ix,:);