Erm, if performance is really bad, it is a good time to think about your choice of algorithm. You have already noted that there is a polygon around each site in the input set - or put differently, the edges that form a polygon have an identical site.
Therefore, something as simple as the following should solve it quite nicely:
Map<Integer, List<GraphEdge>> edgesByPolygon = new HashMap<>();
for (GraphEdge edge : edgesList) {
List<GraphEdge> list = edgesByPolygon.get(edge.site1);
if (list == null) {
list = new ArrayList<>();
edgesByPolygon.put(edge.site1, list);
}
list.add(edge);
list = edgesByPolygon.get(edge.site2);
if (list == null) {
list = new ArrayList<>();
edgesByPolygon.put(edge.site2, list);
}
list.add(edge);
}
for (List<GraphEdge> list : edgesByPolygon.valueSet()) {
// order the edges by adjacency and construct the polygon instance
// (a naive algorithm will do, as the average number of edges is small)
}
This being a nearly O(n) algorithm (instead of your O(n^2)), I estimate this should be about 1000 times faster.