Buildout: use dependencies from system Python

Question

I'm trying to use buildout for a Python package which, when used, depends on 2 extension modules: dbus-python and pygobject. Both modules make buildout fail: dbus-python lacks a setup.py file, while pygobject has one but whose usage is discouraged -- instead configure, make, make install should be used. So, buildout isn't able to setup these dependencies in the development environment.

Here's my buildout.cfg:

[buildout]
develop = .
parts = eggs

[python]
recipe = zc.recipe.eggs
interpreter = python
eggs = foobar

where setup.py for the foobar package contains:

install_requires=['dbus-python', 'pygobject'],

While looking for a solution, I stumbled upon the recipe z3c.recipe.scripts and its ability to utilize system-wide installed eggs. However, when applied to my buildout.cfg ..

[python]
recipe = z3c.recipe.scripts
include-site-packages = true
allowed-eggs-from-site-packages = pygobject, dbus-python
interpreter = python
eggs = foobar    

.. it appears to have no effect (still fails), although both packages (dbus, gobject) are installed in my system Python. The same is true when I remove the allowed-eggs.. line.

My question: Did I got something wrong here on a conceptional level or is there an error in my buildout.cfg?

I know there's zc.recipe.cmmi, a recipe which installs eggs using configure, make, make install. However, simply referencing system Python eggs would be sufficient in my case. I do not need a 100% reproducible environment generated by buildout. Also, dbus-python and pygobject are installed by default on most Linux desktop systems, the environment where foobar is intended to be used.

Answer 1

Thanks to @Rainout's answer and comments I found the actual source of my problem. The problem is not in buildout or my configuration but in the Python bindings for DBus and Gobject: these packages aren't distributed as eggs but as plain packages.

So while these packages can be retrieved via PyPI, they cannot be used in any infrastructure which expects Python packages to be shipped as eggs. In practice this means that the dependencies to these packages must not be listed in setup.py but need to be handled in some other way (if at all).

Answer 2

I haven't gotten the latest 1.5.x buildouts to work with system packages, too. There is one way: pin the versions. That way, zc.buildout 1.5.x will pick it up.

[buildout]
...
versions = versions

[versions]
pygobject = 1.2.3

Alternatively, what I do, you can use the old 1.4.4 buildout (you'll need a special bootstrap.py for that, google it) in combination with osc.recipe.sysegg.

[buildout]
...
parts = 
    ...
    sysegg

[sysegg]
recipe = osc.recipe.sysegg
force-sysegg = true 
eggs =
    dbus-python
    pygobject

I'd personally go for the osc.recipe.sysegg solution, as that's reliable.

Answer 3

You might want to use a CMMI part for each of those poorly behaving python distributions and use the extra-paths option for your python part to make sure the CMMI parts are included in the python path.

Answer 4

The best way I've found to do this is to set include-site-packages to true, then use the mockedeggs recipe to trick setuptools into thinking the eggs are available during installation. The only downside is you don't have much control over what gets used from site-packages; you can set allowed-eggs-from-site-packages to blank to stop eggs being used, but all packages will be available. Anyway, here's a snippet from my buildout:

[buildout]
parts =
    mockedeggs
    myapp

include-site-packages = true
allowed-eggs-from-site-packages =

[myapp]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}

[mockedeggs]
recipe = collective.recipe.mockedeggs
mocked-eggs =
    mapnik2 = 2.0