Ubuntu + virtualenv = a mess? virtualenv hates dist-packages, wants site-packages

Question

Can someone please explain to me what is going on with python in ubuntu 9.04?

I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9) and everything seems to be installed to /usr/local/lib/python2.6/dist-packages

I assume that when installing a package using apt-get, it's placed in /usr/lib/python2.6/dist-packages/ ?

The issue is, there is a /usr/local/lib/python2.6/site-packages as well that just sits there being empty. It would seem (by looking at the path in a virtualenv) that this is the folder virtualenv uses as backup. Thus even thought I omit --no-site-packages, I cant access my local systems packages from any of my virtualenv's.

So my questions are:

1. How do I get virtualenv to point to one of the dist-packages?
2. Which dist-packages should I point it to? /usr/lib/python2.6/dist-packages or /usr/local/lib/python2.6/dist-packages/
3. What is the point of /usr/lib/python2.6/site-packages? There's nothing in there!
4. Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist-packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?
5. Why the hell is this so confusing? Is there something I'm missing here?
6. Where is it defined that easy_install should install to /usr/local/lib/python2.6/dist-packages?
7. Will this affect pip as well?

Thanks to anyone who can clear this up!

Answer 1

I'd be tempted to hack it by making site-packages a link to dist-packages, but I guess this might affect other cases where you want to install some extension other than from the ubuntu dist. I can't think of another answer to 1 except tweaking virtualenv's sources (with both ubuntu and virtualenv being so popular I wouldn't be surprised to find tweaked versions already exist).

Re 2, if you're using /usr/local/bin/python you should use the /usr/local version of the lib (including site-packages) and conversely if you're using /usr/bin/python.

Re 3, there will be something there if you ever install an extension for /usr/bin/python from sources (not via easy_install or from ubuntu's distro).

Re 4, yes, earlier entries on the path take precedence.

Re 5, easy_install is easy only in its name -- it does so much dark magic that it's been carefully kept out of the standard python library despite its convenience because the consensus among us python committers is that deep dark magic for convenience is "easy" only on the surface.

Re 6, I think that's an ubuntu modification to easy_install -- if that's right then it's defined wherever Canonical or other ubuntu maintainers make their collective decisions.

Re 7, sorry, no idea -- I have no reasonably recent ubuntu at hand to check.

Answer 2

I believe Mike Orr's answer from the virtual-env mailing list seems to be the best. Note the OP published this question in both places.

Original content of mail:

Years ago Debian created /usr/local/lib/pythonVERSION/site-packages, and compiled the Python binary to include it in the default search path. Ubuntu followed Debian's lead as it normally does. The Python developers did not like this because you'd get interference with a locally-installed /usr/local/bin/python using the same site-packages directory. Ubuntu finally decided to abandon site-packages and use dist-packages instead, a name which they invented so it wouldn't interfere with anything. The loing story is out there somewhere if you google it, somewhere in the Python bug tracker or distutils SIG or such.

The system works, at least if you use the Ubuntu virtualenv package. Some people have had problems using a locally-installed virtualenv on Ubuntu because the magic sys.path entries weren't being added or something. I'm not sure about --no-site-packages because I never use that option: I run PIL and mysqldb from the Ubuntu packages because it can be hard to compile their C dependencies sometimes. (Need the right header files, Python is ignoring the header files, etc.)

So Ubuntu Python packages go into /usr/lib/pythonVERSION/dist-packages. Or that python-support directory for some reason. Locally-installed Python packages go into /usr/local/lib/pythonVERSION/dist-packages by default. Whenever I install an Ubuntu 9.04 system I run:

$ sudo apt-get install python-setuptools (6.0c9) $ sudo apt-get install python-virtualenv (1.3.3) $ sudo easy_install pip $ sudo pip install virtualenvwrapper

The virtualenvs work fine this way, although I haven't tried --no-site-packages.

I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9)

These versions are both in Ubuntu 9.04, so you're making it harder on yourself by installing them locally.

and everything seems to be installed to /usr/local/lib/python2.6/dist-packages

Yes

I assume that when installing a package using apt-get, it's placed in / usr/lib/python2.6/dist-packages/ ?

Yes

4. Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist- packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/ python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?

sys.path is scanned in order. The only funny thing is that .pth eggs get put earlier or later in the path than some people expect. But if you're using pip for everything it can do (i.e. except to install pip itself, precompiled eggs, and a snapshot of a local directory that's a copy rather than an egg link), you won't have many .pth eggs anyway.

5. Why the hell is this so confusing? Is there something I'm missing here?

It's not well documented. I figured it out by scanning the web.

7. Will this affect pip as well?

Yes, pip will automatically install to /usr/local/lib/pythonVERSION/site-packages. Use "pip install -E $VIRTUAL_ENV packagename" to install into a virtualenv.

Answer 3

You really should not touch Ubuntu's Python installation unless you are building system admin tools, or building something that could be considered to be a new system service.

If you are using Ubuntu to develop or deploy Python applications, always build your own Python from source, tar it up, and use that for deployment. That way you will have all the directories in the right place and virtualenv will work normally. If you will deploy several Python apps on the server, then make your Python live in some place like /home/python or /opt/python or somewhere outside of your home directory. Make sure that you have write permissions for the developers group (users?) so that people can easily add packages.

This also allows you to have two tiers of packages. The ones that are your in-house standard tools can be installed in your Python distro and be part of the tarball that you deploy, and only the app-specific packages would be in a virtualenv.

Do not upgrade or modify the Ubuntu system installed Python.

Answer 4

Well I have a Ubuntu 9.04 and quickly tried setting up a couple sandboxes with site-packages and one without. And things are working fine.

The only difference in the approach I took is I used Ubuntu's python-virtualenv package (1.3.3). And presume that it is tweaked by Ubuntu team to suit Ubuntu setups.

To sum up disable easy_installed virtualenv for a while, use packaged python-virtualenv and see if that meets your expectations.

In fact we use similar setup for production without any problem. Rest is already answered by Alex.

Answer 5

Another way to fix it:
https://stackoverflow.com/a/17265840/202168

Have to remember to do that in each virtualenv where you need it, but doesn't rely on hacks or a special version of virtualenv