Does it make sense to install my Python unit tests in site-packages?

Question

I'm developing my first Python distribution package. My learning curve on Python packaging seems to be leveling off a bit, but I'm still wrestling with a few open questions. One is whether I should cause my unit tests to be installed alongside my code.

I understand it's important to include tests in a source distribution. What I'm wondering is whether I should actually configure them to be installed?

I've seen at least one popular package that appears to do this on purpose (PyHamcrest), and at least one other that appears to do it by accident (behave).

So my (multi-part) question is this:

• Does it ever make sense to install my package unit tests alongside my package code?

• If so, what is the use case? Who would use them and for what? That is, who would use them that wouldn't be perfectly happy to download the source distribution and run python setup.py test instead?

• And how would they use installed unit tests? like import test; test.run() or something like that?

Answer 1

After researching this issue, and until someone more experienced has a minute to weigh in to the contrary, my understanding is that the simple answer is: "No, unit tests should not be installed, only included in the source distribution".

In the handful of cases I found where tests were installed, all turned out to be accidental and it's easier than one might think to make the mistake without noticing it.

Here's how it happens:

1. The packages=find_packages() parameter is used in setup.py so packages can be found without having to list them out explicitly.
2. The test folder is made into a package (by adding __init__.py) so tests can reference the modules they test using relative naming (like from .. import pkg.mod).
3. setuptools installs test as a separate package, alongside the other(s) in the project. Note this means you can execute import test in the python interpreter and it works, almost certainly not what you intended, especially since a lot of other folks use that name for their test directory :)

The fix is to use the setting: packages=find_packages(exclude=['test']) to prevent your test directory from being installed.

Answer 2

In my opinion the right answer is NO but you will find quite a few distributions that install the tests. Tests shouldn't be installed but they should be included in the source distribution. In my opinion in an ideal world testing installed packages should be a task performed by the package manager (pip) and the site-packages directory shouldn't be polluted with test sources.

I've recently researched this topic and gathered information from various sources and found several different ways to structure the directory/package hierarchy of a distribution that contains both library sources and tests. Most of those structures seem to be obsolete and they were invented as attempts to work around the incomplete feature-sets of older distribution systems at the time. Unfortunately a lot of online sources (older blogposts/documentation) are still advertising the outdated methods so it's very easy to find an outdated distribution how-to/tutorial with online search.

Let's assume you have a library called "my_lib" and you want to structure the sources of your distribution. I will show two popular and seemingly outdated ways to structure your distribution and a third way I've found to be the most versatile. The third method may also be outdated but that's the best one I know a the time of posting this answer. ;-)

Method #1

Distributions that (intentionally or unintentionally) install tests usually use this method.

hierarchy

+- my_lib
|  +- __init__.py
|  +- source1.py
|  +- source2.py
|  +- tests
|     +- __init__.py
|     +- test_1.py
|     +- test_2.py
+- setup.py

Method #2

Tests aren't installed but they should be included in the source distribution through the MANIFEST.in file.

hierarchy

+- my_lib
|  +- __init__.py
|  +- source1.py
|  +- source2.py
+- tests
|  +- __init__.py
|  +- test_1.py
|  +- test_2.py
+- setup.py

Method #3 (I prefer this one.)

This is pretty much similar to Method #2 with a little twist (the src dir).

hierarchy

+- src
|  +- my_lib
|     +- __init__.py
|     +- source1.py
|     +- source2.py
+- tests
|  +- __init__.py
|  +- test_1.py
|  +- test_2.py
+- setup.py

setup() call in setup.py

from setuptools import setup, find_packages

setup(
    ...
    packages=find_packages('src'),
    package_dir={'': 'src'},
    ...
)

MANIFEST.in

recursive-include tests *.py

Tests won't be installed but they will be included in the source distribution through our MANIFEST.in.

In case of method #3 you have an src directory that usually contains only a single package that is the root of your lib. Putting the my_lib package into an src directory (directory and not a package so you don't need an src/__init__.py) has the following benefits:

• When you execute setup.py the directory that contains setup.py is implicitly added to the python path. This means that in your setup.py you can accidentally and improperly import stuff from your library if it's package is in the same directory as setup.py. By putting the my_lib package into src we can avoid this problem.

• You can easily use your distributed test sources to test both the distributed library sources and also the installed library:

  • When you run the tests with setup.py test the package_dir={'': 'src'} part of your setup() call guarantees that your tests will see your my_lib library package that you keep in src/my_lib.
  • You can also run tests without setup.py. In the simplest case you can do that with the python -m unittest command. In this case the src dir won't be part of the python path so you can use this method to test the installed version of the library instead of the sources in src.

Answer 3

However I am no expert, I would like to share my opinion.

I would always put tests alongside the code if I expect something might fail depending on the external reasons. Be it bit-order, strange time zones, character coding, 24-bit integers or anything else bizarre you can encounter and have a test for.

Who would not be happy to download the source and run tests? Maybe some debian users that packages are stripped from sources (I know you are talking about python but let me be a little bit general) and your library can occasionally fail due to some strange things in the system.

If your tests ensure internal sanity only, I would skip attaching them, as with out sources they are not worth much, since you will never alter the internals of the library.

Personally, I've heard about a thing failing cause it was moved to some IBM machine which had different bit-ordering. I don't remember if it depended on bit operation or had something pre-computed and cached statically. But it is sometimes wise to check whether you load what you think you saved.

EDIT: Maybe it will be better to rephrase it. I would install tests when you feel there might be portability caveats. I think it is always good to check things when you deploy stuff on a different system.