Xcode CI bot errors during unit testing (Unexpected TestSuiteWillFinish)

StackOverflow https://stackoverflow.com/questions/21999643

Question

We recently set up some Xcode bots with an iOS app to automatically build and test an app I'm working on. If I do just build and analyze, the build process works fine. If I set it to run tests, it ends up with a variable amount of errors that end with Unexpected TestSuiteWillFinish. For example, the most recent run showed:

Run test suite TimeClockTestCase encountered an error (Unexpected TestCaseDidFinish) testParseResponseString encountered an error (Unexpected TestSuiteWillFinish) testGetIntegrationID encountered an error (Unexpected TestSuiteWillFinish)

And the one before that had just one error:

testAddChildObjectTypeTimeClockEvent encountered an error (Unexpected TestSuiteWillFinish)

The integration results show all tests as passed, no matter what errors are displayed. If I run the tests directly via Xcode (instead of running the bot on the server), there are no errors and the same number of tests are passed.

What causes these errors and how can I eliminate them?

These are the logs from the TimeClockTestCase:

Test Suite 'TimeClockTestCase' started at 2014-02-23 23:11:09 +0000
2014-02-23 18:11:09.653 -0500 [TimeClockResponseCommand parseResponseString] [Line 74] W: Unsupported action number "3" in TIMECLOCK response command

Test Case '-[TimeClockTestCase testAddChildObjectTypeTimeClockEvent]' started.
Test Case '-[TimeClockTestCase testAddChildObjectTypeTimeClockEvent]' passed (0.000 seconds).
Test Case '-[TimeClockTestCase testAddChildObjectTypeTimeClockMode]' started.
Test Case '-[TimeClockTestCase testAddChildObjectTypeTimeClockMode]' passed (0.000 seconds).
Test Case '-[TimeClockTestCase testTimeclockEventColl]' started.
Test Case '-[TimeClockTestCase testTimeclockEventColl]' passed (0.000 seconds).
Test Case '-[TimeClockTestCase testTimeclockModeColl]' started.
Test Case '-[TimeClockTestCase testTimeclockModeColl]' passed (0.000 seconds).
Test Suite 'TimeClockTestCase' finished at 2014-02-23 23:11:09 +0000.
Was it helpful?

Solution

I got the same random behavior. I was able to identify that I was making an Asynchronous call, and testing it, was what caused this random Unexpected TestSuiteWillFinish.

As I'm using Kiwi, what I did is to wait a little bit the end of the asynchronous call like indicated here (https://github.com/allending/Kiwi/wiki/Asynchronous-Testing) for those specicfic asynchronous calls.

Hope this help!

OTHER TIPS

We had the same issue of Unexpected TestSuiteWillFinish calls causing tests to abort. In our case, it was due to excessive debug logging. Disabling our logging caused tests to pass reliably again.

That wouldn't seem to be the cause of your issue, but maybe it's a hint?

My offhand guess is that the XCTest runner has some kind of timeout. So check that your test executes in a timely manner?

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top