A bit more detail about what your trying to do might be helpful. That being said, seems like an algorithm for generating the convex hull might be useful. Such as the following, which is probably the best balance of efficiency and ease of implementation:
http://en.wikibooks.org/wiki/Algorithm_Implementation/Geometry/Convex_hull/Monotone_chain
Once you know which points are part of the convex hull, the rest should be a bit more straight forward.