So I actually figured out the answer to this problem and am posting the solution for everyone. The best choice if you are using python, is to just use the mayavi and tvtk packages from Enthought. Mayavi is a gui on top of the C++ VTK libraries. And tvtk is actually a wrapper for python access to VTK objects. So this allows a person to use python GIS packages--like pyshp, Shapely, and others to manipulate GIS objects and then write them to robust and fast mayavi for visualization. At the same time, if you want to stick to C++ then you can still just write your code in C++ using gdal or ogr, etc., and then run your visualization in VTK. This seems a lot easier and more intuitive then trying to run through some other packages like GRASS, QGIS, or ArcGIS.
Here is a good example of this toolset in action. Example