As it turns out, there was a file in the project, .bundle/config, that said:
BUNDLE_PATH: vendor/bundle
I took a look in vender/bundle, and sure enough there were two directories, 1.8.7 and 1.9.1
I'm sure to get my project working with 1.9.3, i would either have to create a new directory there, or remove the bundle path from my .bundle config.
To resolve this, I'm just going to get 1.9.1 installed. O_o