XCTest and NSRunLoop for async tests

Question

I have seen in many places (including Apple Dev Forum) that to test async operations some developers recommend grabbing current run loop and let it run for a while to force async blocks to get called. e.g.

__block id returnedModel = nil;
BOOL result = [binder fetchAndBind:...
                      successBlock:^(id *model) { returnModel = model; }
                        errorBlock:nil];

NSDate *loopUntil = [NSDate dateWithTimeIntervalSinceNow:10.0f];

BOOL isModelReturned = (returnedModel != nil);

while (!isModelReturned && [loopUntil timeIntervalSinceNow] > 0)
{
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:loopUntil];
    isModelReturned = (returnedModel != nil);
}

There are differnet flavors to the above implementation but the concept is the same. Some are using dispatch_group, etc.

Questions:

• Does Apple have any documentation about testing async operations (I couldn't find any)?

• From unofficial sources I have read that unit tests are self-contained in the run loop they are running. So they are not supposed to be treated as above. Is that true? Is this documented by Apple somewhere?

• With Xcode 5.1 the above implenetation or dispatch_group result in EXC_BAD_ACCESS (code=2, address=0xd). Is it because of the concept that unit test are self-contained in their own thread and should not be treated like this?

I have seen problems and side effects with such approaches, particualrly if one or more objects in that test are mocked. Side effects like causing the app to crash so that unit tests can't be completed. For example a method in Class A that its public API takes an NSArray as input crashed because a test mocked an object, let run loop continue, that object then started interacting with Class A and since it was mocked, it passed in a dictionary! -- whereas if run loop was not forced to continue, the object would have been un-mocked later and every test would be happy!

I personally think there is no reason to test async. There is an operation that's running in an async fashion. It is that operation/function that needs to be tested not async.

I am looking for some references or documentation (perferrably from Apple) to clearly talk about async unit tests, whether the run loop of a unit test can be forced to continue, or what is the recommended approach for testing async operations with XCTests.

Thanks!

---

Edit:

In Xcode 6 the XCTest framework ships with async testing macros. I leave this question here for reference.

Answer 1

There are a few misconceptions regarding your example code:

First off, if the completion handler will be executed on a different thread than where the call-site executes, the compiler will create code that is "undefined behavior". This is due to modifying the variable returnedModel in thread "A" and reading the value in thread "M". This is a classic "data race" which produces undefined behavior (please read more in the C and C++ specification).

The __block modifier may alleviate this issue, but I don't believe clang takes special actions here. In the worst case, the thread reading the value (the main thread) never "sees" an update of the value performed through the handler or it reads "garbage".

Another problem with this approach requires a more thorough understanding how Run Loops do actually work. In your example, in the worst case, the run loop's method runMode:beforeDate: will only return when the timeout expired - that is after 10 secs. It may return earlier only if there was an event processed on this mode - possibly unrelated to the test code.

So in short, this approach isn't really suited to accomplish the task. But other "flavors" may indeed work.

According your questions:

Q1: No.

The reason is probably, that XCTest is actually quite old (its just another name for SenTest), and code at the time where it was invented had probably no such fancy stuff like "asynchronous operations", "blocks" and "completion handlers". So there's no built-in solution for this task.

Q2: I don't quite understand this questions. But we might assume that "matchers" (aka "assert something") use exceptions when the test fails. Those require to be executed on the main thread, where there is a catch handler implemented by the underlying test implementation. Maybe XCTest doesn't use exceptions - however, other Unit Test libraries may indeed use exceptions - for example "Cedar". That means, if you execute a completion handler on some queue, and a matcher throws an exception, it MUST be executed on the main thread. (bummer).

Q3: Perhaps the exception issue? But I have no idea. Possibly there's another issue. You may provide more information.

The other "side" effects may be "race conditions" or other issues. But unless you provide more detailed info I'm guessing ;)

Whether or not there is a need to "test async" really depends on what you are actually testing:

For example, if you use a well known third party network library which has a completion handler, would you really want to test whether the handler will be invoked? (Probably not, since you wouldn't want to actually test the network library).

But if you implemented your own asynchronous operation which reports the result via a completion handler, you actually may want to test whether a completion handler will be invoked.