For each side of polygon, calculate line equation (Ax+By+C=0
) and check (put x
and y
into equation and get sign of it), that all points are from one side of it.
EDIT: If travel convex polygon, you will always rotate into one direction (left or right) on every point. Using cross product, you can simply deduce, onto which side (negative or positive) you will rotate next turn. If all cross products of three consecutive points will have equal sign, then your polygon is convex.