For the swapped points to yield the same result, you need a mathematical operation where A op B == B op A
and you need to apply it to both coordinates before adding the result to the hash.
I would try this:
hash = hash * 31 + (
this.Geometry.FromPoint.X.GetHashCode()
+ this.Geometry.ToPoint.X.GetHashCode
);
This line returns the same result, no matter in which order you pass the X coordinate.
Note: If you add/remove lines of the polygon or move end points, then the hash code changes. So you must make sure that the geometry doesn't change as long as such an object is stored in a hash map/set.
If you need to change the geometry, you first have to remove the object from the hash map/set, change the geometry and add it again.
PS: The X
in the last line of your code should be Y
.