Can PHP SimpleTest framework be configured to fail fast?

Question

I'm new to the PHP SimpleTest framework, and I was surprised to see that a failed assertion doesn't halt the test method. In other words, this causes two failure messages in the test report:

function testFoo() {
    $this->assertTrue(true, 'first: %s');
    $this->assertTrue(false, 'second: %s');
    $this->assertTrue(false, 'third: %s');
}

Most of my unit testing experience is with JUnit and NUnit, and they both halt a test method as soon as the first assertion fails. Maybe I'm just used to that, but it seems like the extra failure messages will just be noise. It reminds me of old C compilers that would spew 50 errors because of a missing semicolon.

Can I configure SimpleTest to fail fast, or do I just have to live with a different style?

Answer 1

You could extend/modify the Reporter class so it will exit() after a paintFail().
(No modification of the unittests required)

Or

The assert* functions return a boolean value, So for example:

 $this->assertTrue(false, 'second: %s') or return;

would end the current test function.

PS:
If you're using PHPUnit_TestCase class instead of UnitTestCase, the assert* functions won't return a boolean.

Answer 2

This is not really answering your question, but for what it's worth, PHPUnit always fails fast.

http://www.phpunit.de/manual/current/en/goals.html says:

...the first time a test fails, execution of the test halts, and PHPUnit reports the failure.

Answer 3

The assert methods do return a boolean value for pass or fail, so you can check that and stop the test when something fails.

I don't think this is scalable to every test in a project, but some particularly noisy tests might benefit from it.

Here's a simple example:

function testBar() {
    $pass = $this->assertTrue(true, 'first: %s');
    $pass = $pass && $this->assertTrue(false, 'second: %s');
    $pass = $pass && $this->assertTrue(false, 'third: %s');
}

You could use if statements to wrap larger chunks of code.

Answer 4

I needed this because all of my tests depend on the database being up. IF the database is down I don't really care about a screen with 10,000 failures. Since simpletest prints immediately upon failure and it is just php code you can actually stop testing with a simple die statement.

Here is what my code looks like for my critical "am I connected" test.

function testDatabaseAccess()
{
  $connected = TRUE;
  GLOBAL $clients;
  $connected &= $this->assertTrue(is_object($clients), 'Clients database not connected.');

  GLOBAL $practices;
  $connected &= $this->assertTrue(is_object($practices), 'Practices database not connected.');

  GLOBAL $user;
  $connected &= $this->assertTrue(is_array($user), 'User not defined.');

  GLOBAL $practice;
  $connected &= $this->assertTrue(is_object($practice), 'Practice database not connected.');

  if (!$connected)
  {
    die('Not Connected.');
  }
}

Which produces:

Of course I do not get the summary line but for my use case I don't need or want my summary line. No DB, no point in testing. For the rest of my testing I actually like that it doesn't fail immediately. I usually want to see all my failures in one run.