How to extend xUnit to run each test case as a separate executable

Question

I have created a unit testing framework, pFUnit, that largely follows the current design of JUnit, and am looking for suggestions on how to extend this framework to handle a certain situation. For the curious, this framework pFUnit is written in OO Fortran (yes -- Fortran now has OO capabilities!) and supports distributed programing using MPI. But I think the only pertinent aspect about language choice is that if the SUT actually crashes, the testing framework does as well. Fortunately, that is a relatively rare situation, but it still happens often enough to scream for a solution.

My intent had been to provide an alternate TestRunner that will run each test as a separate executable as an RPC or something similar. The run-time overhead for this can be large, especially when launching MPI repeatedly, so I do not want to make this the default behavior. Unfortunately, when I looked into how to code this approach, I found that TestRunner does not appear to be ideally suited for such an extension as it only manages the run of the top of a nested seriest of TestSuite's.

I can see a klunky way of making it work by having the TestRunner navigate the nested structure, but it would undermine the role of TestSuite in a major way.

Actually, the easiest approach I've come up with is to subclass TestResult. TestResult invokes runBare() on each TestCase, so an extension could simply be to launch a separate executable that just invokes that runBare() method and returns any exceptions. This solution bothers my sense of aesthetics, as it is not the sort of thing for which TestResult ought to be responsible.

I could also add a launch() method to TestCase that checks some global parameter to determine whether to run as a procedure or to launch as a separate executable. This seems inelegant, but is probably not much more difficult than the TestResult extension I mentioned before.

Hopefully this is enough background that a person with deeper understanding of JUnit's design can suggest a better/cleaner approach than the alternatives I've proposed. Or failing that offer me absolution for the design sins I have proposed.

Answer 1

JUnit doesn't do forking as such, it leaves that to the things that invoke JUnit, such as Eclipse or Maven surefire.

Eclipse forks the jvm and then executes in the separate JVM, and communicates the result through a connection to the forked JVM.

Maven surefire has options where the user can control how and when to fork - it has different options for forking, see Maven Surefire - Fork Options and Parallel Test Execution.

One option would be to have a ForkingTestRunner, a TestRunner which forks. Then you could do the equivalent of @RunWith(ForkingTestRunner.class), which forks once per test method.

If this is still too slow, you could fork a runner, and communicate the results back through a socket or something to the monitor process. If the SUT crashes, then the monitor process detects this and fails the test and restarts the runner (but with a cut down list).

This way you'd only have one fork per set of tests, unless the SUT crashes a lot.