Place boost python extension inside package

Question

I have the following boost python code:

#include <boost/python.hpp>

namespace bp = boost::python;

class PyExtTest
{
public:
    std::string get_name() { return m_name; }
    void set_name(std::string const& name) { m_name = name; }

private:
    std::string m_name;
};

BOOST_PYTHON_MODULE(test_ext)
{
    bp::class_<PyExtTest>("pet")
        .add_property("name", &PyExtTest::get_name, &PyExtTest::set_name)
        ;
}

which I compile into the the following directory hierarchy using distutils:

build/lib/test_ext.so

and then use (with PYTHONPATH set to build/lib) as so:

import test_ext
a = test_ext.pet
a.name = 'dog'
print a.name     # dog

When I configure distutils to install test_ext as part of package pkg, python can no longer find the module. I end up with the directory hierarchy:

build/lib/pkg
build/lib/pkg/__init__.py
build/lib/pkg/test_ext.so

from using a setup script like:

test_module = Extension('pkg.test_ext'
                        , define_macros = module_macros
                        , extra_compile_args = module_compiler_flags
                        , sources = ['pkg/test_ext/test.cpp']
                        , include_dirs = module_include_dirs
                        , library_dirs = module_lib_dirs
                        , libraries = module_libs)

setup (name = 'Test'
       , version = '0.1'
       , description = 'Test'
       , packages = ['pkg']
       , ext_modules = [test_module])

but trying to import the module:

import pkg.test_ext

results in the error:

ImportError: No module named test_ext

If I substitute a pure python module for test_ext, then everything works as expected.

I think I need to change the boost code in some manner to indicate that the test_ext extension is inside the pkg package, but I cannot figure out exactly how to do it.

I tried just moving test_ext.so to the correct location in the build/lib directory hierarchy to get the package layout that I wanted, but the only place it is recognized as a module is directly under build/lib. This is different the .py files, which I can import successfully from wherever I place them in the directory hierarchy.

I am using python 2.7.6 and boost 1.55.

Any help is appreciated.

Answer 1

As demonstrated in the Boost.Python Creating Packages tutorial, there is nothing special required between importing a pure python module and a python extension module.

---

Given the following directory structure, where src/pkg/__init__.py is empty and src/test_ext/test_ext.cpp contains the Boost.Python code posted in the question:

.
|-- setup.py
'-- src
    |-- pkg
    |   '-- __init__.py
    '-- test_ext
        └'-- test_ext.cpp

With the following setup.py file containing paths specific to my environment:

from distutils.core import setup, Extension

test_module = Extension('pkg.test_ext',
                        sources=['src/test_ext/test_ext.cpp'],
                        include_dirs=['/usr/local/include'],
                        library_dirs=['/usr/local/lib/boost'],
                        runtime_library_dirs=['/usr/local/lib/boost'],
                        libraries=['boost_python'])

setup(name='Test',
      version='0.1',
      description='Test',
      package_dir={'': 'src'},
      packages=['pkg'],
      ext_modules=[test_module])

The python setup.py build command produces the following directory tree:

.
|-- build
¦   |-- lib.linux-i686-2.7
¦   ¦   '-- pkg
¦   ¦       |-- __init__.py
¦   ¦       '-- test_ext.so
¦   '-- temp.linux-i686-2.7
¦       '-- src
¦           '-- test_ext
¦               '-- test_ext.o
|-- setup.py
'-- src
    |-- pkg
    ¦   '-- __init__.py
    '-- test_ext
        '-- test_ext.cpp

The pkg.text_ext module can be imported by adding the build/lib.linux-i686-2.7 directory path to the Python Module Search Path. For example, either appending the path to:

• the PYTHONPATH environment variable before starting the interpreter
• the sys.path list within the interpreter.

In this example, I have opted to start the interpreter from the directory that contains build and append build/lib.linux-i686-2.7 to sys.path:

>>> import sys
>>> sys.path.append('build/lib.linux-i686-2.7')
>>> import pkg.test_ext
>>> parrot = pkg.test_ext.pet()
>>> parrot
<pkg.test_ext.pet object at 0xb7439a7c>
>>> parrot.name = 'Polly'
>>> print parrot.name
Polly

---

If failures still occur when importing the module, then consider running python with -vv arguments. This will cause python to print verbose tracing messages for each file that is checked when importing. It may be helpful to observe where Python is searching and finding the pkg package, but failing to find the test_ext module. For example, if I start the Python interpreter from within the src directory:

src$ python -vv
# ... some python loading verbose tracing ...
>>> import
# ... get the verbose tracing for the imports import needs out of the way...
  File "<stdin>", line 1
    import
         ^
SyntaxError: invalid syntax
>>> import pkg.test_ext
import pkg # directory pkg
# trying pkg/__init__.i386-linux-gnu.so
# trying pkg/__init__.so
# trying pkg/__init__module.so
# trying pkg/__init__.py
import pkg # from pkg/__init__.py
# wrote pkg/__init__.pyc
# trying pkg/test_ext.i386-linux-gnu.so
# trying pkg/test_ext.so
# trying pkg/test_extmodule.so
# trying pkg/test_ext.py
# trying pkg/test_ext.pyc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named test_ext

One can see that the pkg package was found relative to my current working directory, but it did not locate a test_ext module after trying various extensions.