How do I get nose to discover dynamically-generated testcases?

Question

This is a follow-up to a previous question of mine.

In the previous question, methods were explored to implement what was essentially the same test over an entire family of functions, ensuring testing did not stop at the first function that failed.

My preferred solution used a metaclass to dynamically insert the tests into a unittest.TestCase. Unfortunately, nose does not pick this up because nose statically scans for test cases.

How do I get nose to discover and run such a TestCase? Please refer here for an example of the TestCase in question.

Answer 1

Nose has a "test generator" feature for stuff like this. You write a generator function that yields each "test case" function you want it to run, along with its args. Following your previous example, this could check each of the functions in a separate test:

import unittest
import numpy

from somewhere import the_functions

def test_matrix_functions():
    for function in the_functions:
        yield check_matrix_function, function

def check_matrix_function(function)
    matrix1 = numpy.ones((5,10))
    matrix2 = numpy.identity(5)
    output = function(matrix1, matrix2)
    assert matrix1.shape == output.shape, \
           "%s produces output of the wrong shape" % str(function)

Answer 2

Nose does not scan for tests statically, so you can use metaclass magic to make tests that Nose finds.

The hard part is that standard metaclass techniques don't set the func_name attribute correctly, which is what Nose looks for when checking whether methods on your class are tests.

Here's a simple metaclass. It looks through the func dict and adds a new method for every method it finds, asserting that the method it found has a docstring. These new synthetic methods are given the names "test_%d" %i.

import new
from inspect import isfunction, getdoc

class Meta(type):
    def __new__(cls, name, bases, dct):

        newdct = dct.copy()
        for i, (k, v) in enumerate(filter(lambda e: isfunction(e[1]), dct.items())):
            def m(self, func):
                assert getdoc(func) is not None

            fname = 'test_%d' % i
            newdct[fname] = new.function(m.func_code, globals(), fname,
                (v,), m.func_closure)

        return super(Meta, cls).__new__(cls, 'Test_'+name, bases, newdct)

Now, let's create a new class that uses this metaclass

class Foo(object):
    __metaclass__ = Meta

    def greeter(self):
        "sdf"
        print 'Hello World'

    def greeter_no_docstring(self):
        pass

At runtime, Foo will actually be named Test_Foo and will have greeter, greeter_no_docstring, test_1 and test_2 as its methods. When I run nosetests on this file, here's the output:

$ nosetests -v test.py
test.Test_Foo.test_0 ... FAIL
test.Test_Foo.test_1 ... ok

======================================================================
FAIL: test.Test_Foo.test_0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/rmcgibbo/Desktop/test.py", line 10, in m
    assert getdoc(func) is not None
AssertionError

----------------------------------------------------------------------
Ran 2 tests in 0.002s

FAILED (failures=1)

This metaclass isn't really useful as is, but if you instead use the Meta not as a proper metaclass, but as more of a functional metaclass (i.e. takes a class as an argument and returns a new class, one that's renamed so that nose will find it), then it is useful. I've used this approach to automatically test that the docstrings adhere to the Numpy standard as part of a nose test suite.

Also, I've had a lot of trouble getting proper closure working with new.function, which is why this code uses m(self, func) where func is made to be a default argument. It would be more natural to use a closure over value, but that doesn't seem to work.

Answer 3

You could try to generate the testcase classes with type()

class UnderTest_MixIn(object):

    def f1(self, i):
        return i + 1

    def f2(self, i):
        return i + 2

SomeDynamicTestcase = type(
    "SomeDynamicTestcase", 
    (UnderTest_MixIn, unittest.TestCase), 
    {"even_more_dynamic":"attributes .."}
)

# or even:

name = 'SomeDynamicTestcase'
globals()[name] = type(
    name, 
    (UnderTest_MixIn, unittest.TestCase), 
    {"even_more_dynamic":"attributes .."}
)

This should be created when nose tries to import your test_module so it should work.

The advantage of this approach is that you can create many combinations of tests dynamically.