which and type show different paths to IPython executable in virtualenv, import fails in IPython, succeeds in standard interpreter?

Question

(myvirtualenv)[~] sam@iota $ type ipython  
ipython is hashed (/usr/local/bin/ipython)  
(myvirtualenv)[~] sam@iota $ which ipython  
/Users/sam/.virtualenvs/myvirtualenv/bin/ipython  

regardless of what which says, it executes /usr/local/bin/ipython

(myvirtualenv)[~] sam@iota $ ipython  

WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.  
Python 2.7.5 (default, Sep  6 2013, 17:51:18)  

IPython 1.1.0 -- An enhanced Interactive Python.

In [1]:  import novaclient
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-1-93e5b70cb5b9> in <module>()
----> 1 import novaclient

/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/novaclient/__init__.py in <module>()
     16
     17
---> 18 __version__ = pbr.version.VersionInfo('python-novaclient').version_string()

/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/pbr/version.pyc in version_string(self)
     70         if self.version is None:
     71             parts = []
---> 72             for part in self.release_string().split('.'):
     73                 if part[0].isdigit():
     74                     parts.append(part)

/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/pbr/version.pyc in release_string(self)
     62         
     63         if self.release is None:
---> 64             self.release = self._get_version_from_pkg_resources()
     65
     66         return self.release

/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/pbr/version.pyc in _get_version_from_pkg_resources(self)
     55             # installed into anything. Revert to setup-time logic.
     56             from pbr import packaging
---> 57             return packaging.get_version(self.package)
     58
     59     def release_string(self):

/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/pbr/packaging.pyc in get_version(package_name, pre_version)
    820     if version:
    821         return version
--> 822     raise Exception("Versioning for this project requires either an sdist"
    823                     " tarball, or access to an upstream git repository."
    824                     " Are you sure that git is installed?")

Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?

now forcing it to execute /Users/sam/.virtualenvs/myvirtualenv/bin/ipython

(myvirtualenv)[~] sam@iota $ `which ipython`

Python 2.7.5
Type "copyright", "credits" or "license" for more information.

IPython 1.1.0 -- An enhanced Interactive Python.

In [1]: import novaclient
In [2]: print "all is well here"
all is well here

(myvirtualenv)[~] sam@iota $ echo $PATH
 /Users/sam/.virtualenvs/myvirtualenv/bin:/usr/local/sbin:/usr/local/lib:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
(myvirtualenv)[~] sam@iota $ python

Python 2.7.5 (default, Sep  6 2013, 17:51:18)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.24)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import novaclient
>>> print "all is well here"
all is well here
>>> exit()  

Why do which and type point to different paths? Why, if IPython is installed in my virtualenv, does the system IPython get executed, even though the order of my $PATH would indicate otherwise? In any case, why does import novaclient bomb out with what seems to be a pretty bizarre error?

---

EDIT

pythonpath from erroneous session

(myvirtualenv)[~] sam@iota $ ipython -c 'import sys;sys.path'

['/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages',
 '',
 '/usr/local/bin',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-1.1-py2.7.egg',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Library/Python/2.7/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/usr/local/lib/python2.7/site-packages/IPython/extensions']

pythonpath from successful session

(myvirtualenv)[~] sam@iota $ python -c 'import sys;print sys.path'

['',
 '/Users/sam/.virtualenvs/projectx-dev-2/bin',
 '/Users/sam/.virtualenvs/projectx-dev-2/src/pyrax',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python27.zip',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/plat-darwin',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/plat-mac',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/lib-tk',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/lib-old',
 '/Users/sam/.virtualenvs/projectx-dev-2/lib/python2.7/lib-dynload',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages',
 '/Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/IPython/extensions']

Answer 1

Why do which and type point to different paths?

If your shell has hashed ipython's path, type will give you the hashed path without searching $PATH, while which will ignore the hashed path and do the search.

Assuming your shell is bash, the hash command (with no args) will show you the hashed paths, or you can look at the BASH_CMDS associative array. But you already know that ipython is hashed, because type just told you.

Usually you can fix this by just starting a new shell, using the hash -r command (to wipe the cache without restarting anything else), or hash -d ipython (to remove just ipython from the cache—but make sure you don't also have, e.g., python in the cache). With different shells, the two commands may be different (e.g., in *csh and zsh, it's rehash and rehash ipython, respectively), but starting a new shell should always works.

For more details, see the Command Search and Execution in the bash reference manual, or the bash(1) manpage on your system, or the numerous easier-to-understand blog posts that will show up if you google something like "bash hash cache" (like this one).

Why, if IPython is installed in my virtualenv, does the system IPython get executed

If your shell has hashed ipython's path, ipython will run from the hashed path, without searching $PATH.

Again, starting a new shell, etc., should fix this.

In any case, why does import novaclient bomb out with what seems to be a pretty bizarre error?

Most likely it's getting the PYTHONPATH from your environment, which causes the venv's site-packages and dist-packages to come become the system Python's, and the packages in those locations don't work with the system Python.

You can verify this by printing out sys.path. But the fact that it's complaining about files in /Users/sam/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/ implies that this path must be on sys.path. And (since you're running from a different directory) not just because the default . is picking it up, but because it's there explicitly. So, all that sys.path could tell you is whether it's an absolute or relative path.