How to make macports python use its own packages over another installation's?

Question

I have spent days struggling to get a scientific Python environment running on Mac OS Lion. I tried the SciPack Superpack route, and also various manual installations via pip and easy_install, but still got errors trying to import or use various modules. Based on the advice in this Stackoverflow thread I set up a fresh installation using MacPorts.

However, when I run the macports Python, it is ignoring packages in the macports installation and instead trying to load incompatible packages from the old installation. I am absolutely sure that I am running the newly installed macports Python. I have checked the symlinks and have checked python_select and have launched Python by directly typing the path to the new installation. But when I try to import, say, statsmodels, it pulls in the old version from another directory.

Here are the contents of sys.path (edited for brevity):

    ['',
 '/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg',
 '/Library/Python/2.7/site-packages/statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg',
    ...
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/Library/Python/2.7/site-packages/twilio-3.6.6-py2.7.egg',
 '/Library/Python/2.7/site-packages/six-1.6.1-py2.7.egg',
 '/Library/Python/2.7/site-packages/httplib2-0.9-py2.7.egg',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
 '/Library/Python/2.7/site-packages']

The macports installation is in /opt/local, while the older installation is under /Library/Python. As you can see, the older packages are higher up in the list, which means that they have higher priority.

Environmental variable PYTHONPATH is empty. If I do put anything into PYTHONPATH, that appears in sys.path after the /Library entries but before the /opt entries. So it does not solve the problem.

If I invoke the new python with the -S option, sys.path becomes:

    ['',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']

That succeeds in getting rid of the alien site packages entries, but it also nukes macports' site-packages entries so then I can't load anything.

I believe that the culprit is a file called /Library/Python/2.7/site-packages/easy-install.pth , with the following contents (again edited for brevity):

import sys; sys.__plen = len(sys.path)
./mrjob-0.4.3_dev-py2.7.egg
    ...
./statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg
    ...
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
./twilio-3.6.6-py2.7.egg
./six-1.6.1-py2.7.egg
./httplib2-0.9-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

If I rename that file before launching macports python, python no longer is adding these alien packages to its sys.path. Now sys.path looks a lot like it does when I use "python -S" option except it has these additional entries at the end:

 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
 '/Library/Python/2.7/site-packages'

Now I am able to load macports' own packages, say for example statsmodels. By checking statsmodels.__file__ I have confirmed that it is importing the local package and not the alien package.

However, I consider this a workaround/kludge rather than a real solution. A solution should make it behave as expected, which is: Macports' Python, launched from /opt/local/... installation, should prioritize packages installed via macports into the /opt/local/... directory tree and go looking elsewhere only if the module does not exist locally. So for example I would expect the /opt/local entries to come first in sys.path, with the /Library entries further down the list.

It seems like this should be the default behavior and I've seen a lot of comments here on Stackoverflow that just assert that this is how Macports Python behaves. So...how do I make it so?

Answer 1

There are at least two separate root causes for the problem you are seeing. One is that non-system OS X framework builds of Python 2.7, including MacPorts Python 2.7, deliberately include the system Python site-packages location, /Library/Python/2.7/site-packages, in sys.path, normally at the end of sys.path after that instance's own site-packages directory. This has been a debated feature in upstream Python and there is an open MacPorts issue to remove it (https://trac.macports.org/ticket/34763), the argument being that it is better to keep the system Python and the MacPorts Python totally separate.

The second root cause is the behavior of the original setuptools package and its easy_install command as supplied with OS X. As you have discovered, it does some fancy manipulation of sys.path by some magic tricks with .pth files, including easy-install.pth, to ensure that the packages that have been installed using easy_install show up first in sys.path and override other installed versions of those packages. Also as you have found, one way - and the easiest way - to remove that behavior is to delete the easy-install.pth file in /Library/Python/2.7/site-packages. That is assuming you don't want to use any of the packages installed there with the Apple-supplied system Python.

The long-term strategy to avoid this problem in the future is to make sure you do not use the Apple-supplied easy_install or easy_install-2.7 commands found in /usr/bin. They will install packages to /Library/Python for use by the system Python and will create or update easy-install.pth. In general, you should avoid use of easy_install all together. Its modern replacement is pip which provides better control and avoids tricks with .pth files. If MacPorts doesn't already provide a port for the Python package you want to install, usually of the form py27-xxxx, install and use the MacPorts py27-pip port instead of easy_install:

/opt/local/bin/pip-2.7 install xxxx

You can also use the usual MacPorts features like port select pip pip27.