Current state of python namespace packages

Question

I would like to have several python sub modules inside a main module, but I want to distribute them as separated python packages. So package A should provide 'my_data.source_a', package B should provide 'my_data.source_b', ... and so on.

I found out that I have to use a namespace package for this, but trying to figuring out the details, I found multiple PEPs covering that problem. PEP 420 seems to be the latest one, which builds upon PEP 402 and PEP 382.

To me it's not clear what the status of the different PEPs an implementations is. So my question is: Is http://pythonhosted.org/distribute/setuptools.html#namespace-packages still the way to go or how should I build my namespace package?

Answer 1

The latest version of Python which is Python 3.7 uses the native namespace packages approach to create namespace packages which are defined in PEP 420.

There are currently three different approaches to creating namespace packages:

1. Use native namespace packages. This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip.
2. Use pkgutil-style namespace packages. This is recommended for new packages that need to support Python 2 and 3 and installation via both pip and python setup.py install.
3. Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe.

Reference: Packaging namespace packages

Answer 2

The Python documentation has a good description of the three ways of creating namespace packages in Python, including guidelines for when to use each of the three methods. Furthermore, this topic is discussed in great depth in a different StackOverflow thread which has a good accepted answer. Finally, if you are someone who would rather read code than documentation, the sample-namespace-packages repo contains examples of namespace packages created using each of the three available methods.

In brief, if you intend your packages to work with Python versions 3.3 and above, you should use the native namespace packages method. If you intend your packages to work with older versions of Python, you should use the pkgutil method. If you intend to add a namespace package to a namespace that is already using the pkg_resources method, you should continue to use method.

---

With native namespace packages, we can remove __init__.py from both packages and modify our setup.py files to look as follows:

# setup.py file for my_data.source_a
from setuptools import setup, find_namespace_packages

setup(
    name="my_data.source_a",
    version="0.1",
    packages=find_namespace_packages(include=['my_data.*'])
)

# setup.py file for my_data.source_b
from setuptools import setup, find_namespace_packages

setup(
    name="my_data.source_b",
    version="0.1",
    packages=find_namespace_packages(include=['my_data.*'])
)

We need to add the include=['my_data.*'] argument because, by default find_namespace_packages() is rather lenient in the folders that it includes as namespace packages, as described here.

This is the recommended approach for packages supporting Python 3.3 and above.

---

With pkgutil-style namespace packages, we need to add the following line to the my_data.__init__.py files in each of our packages:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

This is the approach used by the backports namespace, and by different packages in the google-cloud-python repo, and it is the recommended approach for supporting older versions of Python.