Good way to collect programmatically generated test suites in nose or pytest

Question 1

This turned out to be a mess: Nose pretty much exclusively uses the TestLoader.load_tests_from_names function (it's the only function tested in unit_tests/test_loader) so since I wanted to actually load things from an arbitrary python object I seemed to need to write my own figure out what kind of load function to use.

Then, in addition, to correctly get things to work like the nosetests script I needed to import a large number of things. I'm not at all certain that this is the best way to do things, not even kind of. But this is a stripped down example (no error checking, less verbosity) that is working for me:

import sys
import types
import unittest

from nose.config import Config, all_config_files
from nose.core import run
from nose.loader import TestLoader
from nose.suite import ContextSuite
from nose.plugins.manager import PluginManager

from myapp import find_test_objects

def load_tests(config, obj):
    """Load tests from an object

    Requires an already configured nose.config.Config object.

    Returns a nose.suite.ContextSuite so that nose can actually give
    formatted output.
    """

    loader = TestLoader()
    kinds = [
        (unittest.TestCase, loader.loadTestsFromTestCase),
        (types.ModuleType, loader.loadTestsFromModule),
        (object, loader.loadTestsFromTestClass),
    ]
    tests = None
    for kind, load in kinds.items():
        if isinstance(obj, kind) or issubclass(obj, kind):
            log.debug("found tests for %s as %s", obj, kind)
            tests = load(obj)
            break

    suite = ContextSuite(tests=tests, context=obj, config=config)

def main():
    "Actually configure the nose config object and run the tests"
    config = Config(files=all_config_files(), plugins=PluginManager())
    config.configure(argv=sys.argv)

    tests = []
    for group in find_test_objects():
        tests.append(load_tests(config, group))

    run(suite=tests)

Question 2

For nose you can have both tests in place and select which one to run using attribute plugin, which is great for selecting which tests to run. I would keep both tests and assign attributes to them:

from nose.plugins.attrib import attr

@attr("safe")
class SafeTests(unittest.TestCase):
    # snip 20 test functions

class BombTests(unittest.TestCase):
    # snip 10 different test cases

For you production code I would just call nose with nosetests -a safe, or setting NOSE_ATTR=safe in your os production test environment, or call run method on nose object to run it natively in python with -a command line options based on your TARGET:

import sys
import nose

if __name__ == '__main__':
    module_name = sys.modules[__name__].__file__
    argv = [sys.argv[0], module_name]
    if TARGET == 'prod':
        argv.append('-a slow')

    result = nose.run(argv=argv)

Finally, if for some reason your tests are not discovered you can explicitly mark them as test with @istest attribute (from nose.tools import istest)

Question 3

If your question is, "How do I get pytest to 'see' a test?", you'll need to prepend 'test_' to each test file and each test case (i.e. function). Then, just pass the directories you want to search on the pytest command line and it will recursively search for files that match 'test_XXX.py', collect the 'test_XXX' functions from them and run them.

As for the docs, you can try starting here.

If you don't like the default pytest test collection method, you can customize it using the directions here.

Question 4

If you are willing to change your code to generate a py.test "suite" (my definition) instead of a unittest suite (tech term), you may do so easily. Create a file called conftest.py like the following stub

import pytest

def pytest_collect_file(parent, path):
    if path.basename == "foo":
        return MyFile(path, parent)

class MyFile(pytest.File):
    def collect(self):
        myname="foo"
        yield MyItem(myname, self)
        yield MyItem(myname, self)

class MyItem(pytest.Item):
    SUCCEEDED=False
    def __init__(self, name, parent):
        super(MyItem, self).__init__(name, parent)

    def runtest(self):
        if not MyItem.SUCCEEDED:
            MyItem.SUCCEEDED = True
            print "good job, buddy"
            return
        else:
            print "you sucker, buddy"
            raise Exception()

    def repr_failure(self, excinfo):
        return ""

Where you will be generating/adding your code into your MyFile and MyItem classes (as opposed to the unittest.TestSuite and unittest.TestCase). I kept the naming convention of MyFile class that way, because it is intended to represent something that you read from a file, but of course you can basically decouple it (as I've done here). See here for an official example of that. The only limit is that in the way I've written this foo must exist as a file, but you can decouple that too, e.g. by using conftest.py or whatever other file name exist in your tree (and only once, otherwise everything will run for each files that matches -- and if you don't do the if path.basename test for every file that exists in your tree!!!)

You can run this from command line with

py.test -whatever -options

or programmactically from any code you with

import pytest
pytest.main("-whatever -options")

The nice thing with py.test is that you unlock many very powerful plugings such as html report