It does not work because the dist function when called with two arguments works like this:
Z = dist(W,P) takes an SxR weight matrix and RxQ input matrix and
returns the SxQ matrix of distances between W's rows and P's columns. dist(P',P) returns the same result as dist(P).
That is, if you do this:
a = [ 1 2 3 ;
11 12 13]
b = a(2,:) % Then b = [11 12 13]
...and call:
dist(b, a)
It will try to compute the distance between b's rows (in this case, only a row with three numbers, that is, a 3D point) and a's columns (each column has two numbers, that is, a 2D point). Measuring a distance between them makes no sense.
The reason it worked on your first example was because the matrix was square (2x2). Therefore, you're computing distances between a row (2D) to the other columns (also 2D).