failing a XCTestCase with assert without the test continuing to run but without stopping other tests

Question

I'm trying to test my application using the XCTest framework.

I want my single test case to fail if some logical condition holds (using an assertion). I don't want the rest of the code in the test case to run, because this might lead to problems (access to null pointers, for example) I also want the rest of the test case to run normally, and just the failed test to be marked as failed.

I've noticed XCTestCase has a property called continueAfterFailure. However, setting it to YES caused the failed test to continue executing lines after the assertion, and setting it to NO caused the rest of the tests not to run at all.

Is there a solution to this issue?

Answer 1

Pascal's answer gave me the idea to achieve this properly. XCTool now behaves like OCUnit when an assertion fails: the execution of the test case is aborted immediately, tearDown invoked and the next test case is run.

Simply override the method invokeTest in your base class (the one that inherits from the XCTestCase class):

- (void)invokeTest
{
    self.continueAfterFailure = NO;

    @try
    {
        [super invokeTest];
    }
    @finally
    {
        self.continueAfterFailure = YES;
    }
}

That's it!

Answer 2

The easiest way is to add:

continueAfterFailure = false

into setUp() method. So it will look like this:

Swift

override func setUp() {
    super.setUp()

    continueAfterFailure = false
}

Objective-C

 - (void)setUp {
    [super setUp];

    [self setContinueAfterFailure:NO];
}

Answer 3

One option would be to check the condition normally, then fail and return from the test if it is false.

Something like this:

if (!condition) {
  XCFail(@"o noes");
  return;
}

You could wrap this up in a helper macro to preserve readability.

BDD test libraries like Kiwi are more elegant for this sort of thing, as they make it easier to share setup between many tests which leads to fewer assertions per test.

Answer 4

I am able to use continueAfterFailure and let the other tests run by using this pattern:

self.continueAfterFailure = NO;
@try
{
    // Perform test code here
}
@finally
{
    self.continueAfterFailure = YES;
}

Answer 5

In Swift projects, I use a helper function (defined in a shared superclass of all my tests which itself extends XCTestCase):

/// Like `XCTFail(...)` but aborts the test.
func XCTAbortTest(_ message: String, 
                  file: StaticString = #file, line: UInt = #line
                 ) -> Never {
    self.continueAfterFailure = false
    XCTFail(message, file: file, line: line)
    fatalError("never reached")
}

As the comment suggests, the call to fatalError is never actually executed; XCTFail aborts the test in an orderly fashion (tearDown is called, next test runs, etc.). The call is only there to trick the compiler into accepting Never as return type since XCTFail returns Void (it does return if continueAfterFailure == true).

Note that self.continueAfterFailure is reset to the default true for every test method. You can also make that explicit in setUp().