A simple approach would be to compute first the convex hull of the points using convhull. It will return the points in a cyclic order (counterclockwise).
K = convhull([x1, x2, x3, x4], [y1, y2, y3, y4])
It will return something like that:
K =
1
3
4
2
1
Meaning point [x1,y1]
is the first, then comes [x3,y3]
, then [x4,y4]
and so on (first point is duplicated at the end to close the hull, so you can ignore that).
After that you can compute the angles with no problems.