how to add dozen of test cases to a test suite automatically in python

Question

i have dozen of test cases in different folders. In the root directory there is a test runner.

unittest\
  package1\
    test1.py
    test2.py
  package2\
    test3.py
    test4.py
  testrunner.py

Currently I added the four test cases manually into a test suite

import unittest
from package1.test1 import Test1
from package1.test2 import Test2
from package2.test3 import Test3
from package2.test4 import Test4

suite = unittest.TestSuite()
suite.addTests(unittest.makeSuite(Test1))
suite.addTests(unittest.makeSuite(Test2))
suite.addTests(unittest.makeSuite(Test3))
suite.addTests(unittest.makeSuite(Test4))

result = unittest.TextTestRunner(verbosity=2).run(suite)
if not result.wasSuccessful():
  sys.exit(1)

How to let test runner test all test cases automatically? Such as:

for testCase in findTestCases():
  suite.addTests(testCase)

Answer 1

The above modules are good but NoseTests can be funny when trying to enter in parameters and this is also faster and fit's into another module nicely.

import os, unittest

class Tests():   

    def suite(self): #Function stores all the modules to be tested


        modules_to_test = []
        test_dir = os.listdir('.')
        for test in test_dir:
            if test.startswith('test') and test.endswith('.py'):
                modules_to_test.append(test.rstrip('.py'))

        alltests = unittest.TestSuite()
        for module in map(__import__, modules_to_test):
            module.testvars = ["variables you want to pass through"]
            alltests.addTest(unittest.findTestCases(module))
        return alltests

if __name__ == '__main__':
    MyTests = Tests()
    unittest.main(defaultTest='MyTests.suite')

If you want to add results to a log file add this at the end instead:

if __name__ == '__main__':
    MyTests = Tests()
    log_file = 'log_file.txt'
    f = open(log_file, "w") 
    runner = unittest.TextTestRunner(f)
    unittest.main(defaultTest='MyTests.suite', testRunner=runner)

Also at the bottom of the modules you are testing place code like this:

class SomeTestSuite(unittest.TestSuite):

    # Tests to be tested by test suite
    def makeRemoveAudioSource():
        suite = unittest.TestSuite()
        suite.AddTest(TestSomething("TestSomeClass"))

        return suite

    def suite():
        return unittest.makeSuite(TestSomething)

if __name__ == '__main__':
    unittest.main()

Answer 2

In my opinion you should switch to unittest2 or other test frameworks with discovery features. Discovery tests is a really sane way to run them.

Most known are:

• nosetests
• py.test

For example with nosetest is sufficient to run nosetests from the project root directory and it will discover and run all the unit tests it find. Pretty simple.

Notice also that unittest2 will be included in python 2.7 (and backported till 2.4 I guess).

Answer 3

What I did is a wrapper script which running separate test files:

Main wrapper run_tests.py:

#!/usr/bin/env python3
# Usage: ./run_tests.py -h http://example.com/ tests/**/*.py
import sys, unittest, argparse, inspect, logging

if __name__ == '__main__':
    # Parse arguments.
    parser = argparse.ArgumentParser(add_help=False)
    parser.add_argument("-v", "--verbose",  action="store_true", dest="verbose",  help="increase output verbosity" )
    parser.add_argument("-d", "--debug",    action="store_true", dest="debug",    help="show debug messages" )
    parser.add_argument("-h", "--host",     action="store",      dest="host",     help="Destination host" )
    parser.add_argument('files', nargs='*')
    args = parser.parse_args()

    # Load files from the arguments.
    for filename in args.files:
        exec(open(filename).read())

    # See: http://codereview.stackexchange.com/q/88655/15346
    def make_suite(tc_class):
        testloader = unittest.TestLoader()
        testnames = testloader.getTestCaseNames(tc_class)
        suite = unittest.TestSuite()
        for name in testnames:
            suite.addTest(tc_class(name, cargs=args))
        return suite

    # Add all tests.
    alltests = unittest.TestSuite()
    for name, obj in inspect.getmembers(sys.modules[__name__]):
        if inspect.isclass(obj) and name.startswith("FooTest") and len(name) > len("FooTest"):
            alltests.addTest(make_suite(obj))

    # Run tests.
    result = unittest.TextTestRunner(verbosity=2).run(alltests)
    sys.exit(not result.wasSuccessful())

Then another wrapper for tests:

class FooTest(unittest.TestCase):
    def __init__(self, *args, cargs=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.vdisplay = Xvfb(width=1280, height=720)
        self.vdisplay.start()
        self.args=cargs
        self.log=logging

    def setUp(self):
        self.site = webdriver.Firefox()

    def kill(self):
        self.vdisplay.stop()

Then each test in separate files would look like:

import sys, os, unittest
from FooTest import FooTest

class FooTest1(FooTest):

    def test_homepage(self):
        self.site.get(self.base_url + "/")
        log.debug("Home page loaded.")

Then you can easily run tests from shell like:

$ ./run_tests.py -h http://example.com/ test1.py test2.py

You can use wildcard to specify all files within certain directories, or use a new globbing option (**) to run all tests recursively (enable by shopt -s globstar).