First, let us illustrate what you want (if I got it right). Suppose you have two polygons, one of them has an edge (a, b)
which intersects with an edge (s, r)
of the other one. These polygons also have a clock-wise orientation, so you know the next vertex after b
, and the next vertex after r
. Since the edges crosses, you remove them both, and add four new ones. The new ones you add are: (a, r)
, (r, next(b))
; (s, b)
, (b, next(r))
. So you again have two polygons. This is illustrated in the following figure. Note that by initially removing only two edges (one from each polygon), all the crossing were resolved.
Speeding the trivial implementation of O(n^2)
per iteration is not entirely easy, and 500 points per polygon is a very small amount to be worried about. If you decide that you need to improve this time, my initial suggestion would be to use the Bentley-Otmann algorithm in some smart way. The smart way involves running the algorithm, then when you find an intersection, you do the procedure above to eliminate the intersection, and then you update the events that guide the algorithm. Hopefully, the events to be handled can be updated without rendering the algorithm useless for this situation, but I don't have a proof for that.