Python C-ext namespace mixed with regular python submodules?

Question

This question has some resemblance with:

Nested Python C Extensions/Modules?

Only with a slight twist. Here I'm not trying to mix two C-exts, but one C-ext and a regular python submodule instead.

Is there a way for a C-extension to share the module namespace between the symbols "module.so" and those present in a submodule?

My module structure looks like this:

facs/
    facs/
      __init__.py
      setup.py
      facs.so
      [*.c files]
      utils/
        __init__.py
        galaxy.py

If I remove "utils" from the hierarchy, I can import facs and see the facs.so methods:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove']

But when I put the utils submodule back and try to import the different parts, one namespace seems to mask the other (utils masks the symbols exported by facs.so):

>>> import facs
>>> dir(facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
>>> import facs.utils
>>> facs.utils.galaxy.rsync_genomes("phix")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'galaxy'
>>> from facs.utils import galaxy
>>> galaxy.rsync_genomes("phix")
'Hello world'

As you see, after dir(facs), build, query and remove are gone and galaxy does not get imported properly unless I do a from facs.utils import galaxy instead of re-using the initial import statement and simply accessing via facs.utils.galaxy.rsync_genomes() directly.

To sum up, my intended use of this module would be to do:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove'
, 'utils'] <--- (Directly accessible from "facs")
>>> facs.utils.galaxy.rsync_genomes("phix")
'Hello world'

The (currently under development) code is in:

https://github.com/brainstorm/facs/tree/develop

In case someone wants to try it out by themselves. I'm using virtualenvs and my $PYTHONPATH seems correct:

/home/roman/.venvburrito/lib/python:
/home/roman/.virtualenvs/py27/lib/python2.7/site-packages

And the installation seems to be successful too:

cd ~/.virtualenvs/py27/lib/python2.7/site-packages/facs-2.0dev-py2.7.egg/
(py27)$ ls
EGG-INFO  facs.py  facs.pyc  facs.so  utils/

It seems that no __init__.py file is actually copied to the top level directory but touching it there or not does not affect the importing behavior described above.

Any ideas? Thanks in advance!

Answer 1

Be carefull you are importing the so file first, and not the facs package. Are you running your python console over facs package source directory ?

Try to go outside the package directory and everything will make sense.

Also, next import doesn't say that all modules behind your facs directory will be imported:

>> import facs
>> dir (facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

You have to set your imports in each __init__.py file, for example if you want to export some simbols from your .so file when you import facs, you should have the following contents in your __init__.py file:

>> from _facs import function_name, function_name .....

And then, when you import facs, it will import those functions for you.

Take the same pattern for your sub-package utils.

I would also recommend to rename your facs.so file to _facs.so file to avoid name collisions between package name and module name.