If I understand you correctly, you want to reorder some vertices such that all the lines drawn between subsequent points form a closed, non-overlapping contour.
This can be accomplished by re-ordering your vertices in a (counter)clockwise fashion about the centre of mass of all the points. In 2D, this can most easily be accomplished by sorting the output of atan2
:
%// Compute centre of mass
r_COM = sum([X, Y]) / numel(X);
%// Sort all vertices by angle
[~, I] = sort(atan2(Y - r_COM(2), X - r_COM(1)));
%// Plot the new contour
hold on, plot(X([I; I(1)]),Y([I; I(1)]), '.-k', 'linewidth', 2);
Results: