Since the bottleneck of this use-case appears to be the spatial queries on the data, I would go for the approach where you store the coordinates in a highly optimized data structure for spatial queries and then you have a dictionary where you can retrieve the other features of the points on demand. High performance specialized libraries ie, boost graph and CGAL for computational geometry follow similar philosophies (for example, see property maps in boost http://www.boost.org/doc/libs/1_37_0/libs/graph/doc/using_property_maps.html)
With regards to the appropriate data structure, possibly SciPy has something that suits you http://docs.scipy.org/doc/scipy/reference/spatial.html KDTree would be an option for nearest neighbour queries. Pandas can store the data yes, but it does not have specialized spatial indexing support.