n = size(x,1);
dist = squareform(pdist(x,'cosine')); %// distance matrix
dist(1:n+1:end) = inf; %// self-distance doesn't count
[~, y] = sort(dist,2);
y = y(:,1:n-1);
To save memory, you can work in chunks using pdist2
instead of pdist
:
n = size(x,1);
m = 100; %// chunk size. As large as memory allows. Divisor of n
y = NaN(n,n-1); %// pre-allocate results
for ii = 0:m:size(x,1)-1
ind = ii+(1:m); %// current chunk: these rows
dist_chunk = pdist2(x(ind,:),x,'cosine'); %// results for this chunk
[~, y_chunk] = sort(dist_chunk,2);
y(ind,:) = y_chunk(:,2:end); %// fill results, except self-distance
end