Looks like it's because gdal has some issues with .NET 4.0. It seems to work fine with .NET 3.5 and that's all we need for our project. We might be using an old version of gdal as well. http://trac.osgeo.org/gdal/ticket/4049
It would be nice to know why it runs fine within VS and generates an exception outside though.