Below are the steps I followed.
The 3D planar polygon can be rotated to 2D plane using the known formulas. Use the one under the section Rotation matrix from axis and angle.
Then as indicated by @Glenn, an internal points needs to be calculated to find the angles. I take that internal point as the
mean
of the vertex locations.Using the x-axis as the reference axis, the angle, on a
0
to2pi
scale, for each vertex can be calculated usingatan2
function as explained here.The non-negative angle measured counterclockwise from vector a to vector b, in the range
[0,2pi]
, ifa = [x1,y1]
andb = [x2,y2]
, is given by:angle = mod(atan2(y2-y1,x2-x1),2*pi);
Finally, sort the angles,
[~,XI] = sort(angle);
.