How to test same assertion for large amount of data

Question

I am using python unittest module to do a number of tests; however, it is very repetitive.

I have a lot of data that I want to run through the same test over and over, checking if correct. However, I have to define a test for every one.

For instance I want to do something similar to this. I know I could do it using a generator (found it in a previous thread here). But are there alternatives, maybe even using a different testing module?

Any suggestions would be great.

import unittest

class TestData(unittest.TestCase):
    def testNumbers(self):
        numbers = [0,11,222,33,44,555,6,77,8,9999]
        for i in numbers:
            self.assertEqual(i, 33)

Answer 1

In another post I stumbled accross Nose Tests It is more suited to data driven testing.

class Test_data():
    def testNumbers():
        numbers = [0,11,222,33,44,555,6,77,8,9999]
        for i in numbers:
            yield checkNumber, num

def checkNumber(num):
    assert num == 33

The above code does the exact same thing as my first post. No imports are needed, just write a python class.

You execute the tests by typing:

nosetests filename

Answer 2

Sample code for solution suggested by Bill Gribble could look like this:

import unittest

class DataTestCase(unittest.TestCase):
    def __init__(self, number):
        unittest.TestCase.__init__(self, methodName='testOneNumber')
        self.number = number

    def testOneNumber(self):
        self.assertEqual(self.number, 33)

    def shortDescription(self):
        # We need to distinguish between instances of this test case.
        return 'DataTestCase for number %d' % self.number


def get_test_data_suite():
    numbers = [0,11,222,33,44,555,6,77,8,9999]
    return unittest.TestSuite([DataTestCase(n) for n in numbers])

if __name__ == '__main__':
    testRunner = unittest.TextTestRunner()
    testRunner.run(get_test_data_suite())

Answer 3

You may want to consider using the unittest.TestSuite class, which will allow you to dynamically construct a set of unittest.TestCase instances which will get run separately. Your unittest.TestCase subclass should define just one test method, with the class accepting a construction parameter passing in the value to test against for that particular instance.

Answer 4

The ddt library was built to solve exactly what you are asking for unittest[*].

For example:

import ddt
import unittest

@ddt.ddt
class EvalTests(unittest.TestCase):

    @ddt.data(
            ('1', 1),
            ('1 == 1',  True),
            ('1 == 2',  False),
            ('1 + 2',   4),  ## This will fail
    )
    def test_eval_expressions(self, case):
        expr, exp_value = case
        self.assertEqual(eval(expr), exp_value)

And when you run it, you get 4 TestCases instead of just one:

$ python -m unittest  -v  test_eval.py
test_eval_expressions_1___1___1_ (test_eval.EvalTests) ... ok
test_eval_expressions_2___1__1___True_ (test_eval.EvalTests) ... ok
test_eval_expressions_3___1__2___False_ (test_eval.EvalTests) ... ok
test_eval_expressions_4___1_2___4_ (test_eval.EvalTests) ... FAIL

======================================================================
FAIL: test_eval_expressions_4___1_2___4_ (test_eval.EvalTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python/lib/site-packages/ddt.py", line 129, in wrapper
    return func(self, *args, **kwargs)
  File "/Work/test_eval.py", line 15, in test_eval_expressions
    self.assertEqual(eval(expr), exp_value)
AssertionError: 3 != 4

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1)

Notice that ddt tries to come up with names for the generated TCs.

Install it with pip:

pip install ddt

[*] The same solution for the pythonic pytest framework (pytest.mark.parametrize) is integrated into the core tool, and is worth switching to pytest just for this feature alone.

Answer 5

The problem with running assertions in a loop is that, if one of the assertions fails, you don't know which value caused it (in your example, it would fail on 0, but you don't know that until you debug). On the other hand, repeating self.assertEqual(i, 33) is an even worse idea, because it introduces code duplication.

What I do in my test is to create a simple, shortly-named internal function inside the test and call it with different arguments. So your function would look like this:

import unittest

class TestData(unittest.TestCase):
    def testNumbers(self):
        def eq(i):
            self.assertEqual(i, 33)
        eq(0)
        eq(11)
        eq(222)
        eq(33)
        eq(44)
        eq(555)
        ... 

This way, when the assertion fails for 0, you immediately see it on the stack trace printed by the unittest module.

Answer 6

As of Python 3.4 you can use unittest.TestCase.subTest(msg=None, **params) context manager (documentation). This will allow you to achieve what you want by adding just one statement.

Here is your example modified to use subTest()

import unittest

class TestData(unittest.TestCase):
    def testNumbers(self):
        numbers = [0, 11, 222, 33, 44, 555, 6, 77, 8, 9999]
        for i in numbers:
            with self.subTest(i=i):  # added statement
                self.assertEqual(i, 33)

Answer 7

Spin-off of this answer, which didn't quite work for me. Where I'm not dealing with large amounts of data, I did need to run same tests with different inputs. The following tests are using create_a and create_b methods which I want to customize.

The requirement is to run both tests with the same customization.

class Tests(unittest.TestCase):

    def test_a_uses_b(self):
        a = create_a()
        b = create_b()
        a.b = b
        self.assertIs(b.a, a)

    def test_b_uses_a(self):
        a = create_a()
        b = create_b()
        b.a = a
        self.assertIs(a.b, b)

Instantiating TestSuite and TestCase myself, bypassing the test loader, resulted in an error because it expected a single method, called runTest.

The result was this:

class Tests(unittest.TestCase):

    def __init__(self, create_a, create_b):
        super().__init__()
        self.create_b = create_b
        self.create_a = create_a

    def test_a_uses_b(self):
        a = self.create_a()
        b = self.create_b()
        a.b = b
        self.assertIs(b.a, a)

    def test_b_uses_a(self):
        a = self.create_a()
        b = self.create_b()
        b.a = a
        self.assertIs(a.b, b)


class TestPair1(Tests):
    def __init__(self):
        super().__init__(create_a1, create_b1)


class TestPair2(Tests):
    def __init__(self):
        super().__init__(create_a2, create_b2)