Cucumber - rerun failed tests in Jenkins

Question

we are running a set of Frank/cucumber tests through a jenkins server to test an iOS app.

The tests run locally just fine, and also when run manually on the jenkins server. However, when going through jenkins, we get occasional errors causing the build to fail, which then work fine when we run jenkins again (i.e. press the 'Build now' button) without changing anything.

we run the following code to run the tests:

cucumber features/ipad --tags ~@ignore

I then added the rerun parameter to dump the failed tests into a text file:

-f rerun -o rerun.txt

and then ran cucumber rerun.txt straight afterwards, so ithe whole comman looks like this:

cucumber features/ipad --tags ~@ignore -f rerun -o rerun.txt; cucumber @rerun.txt

This works fine, it catches the failing tests, and reruns them again after the other tests.

However, jenkins still marks the build as a fail, even though the rerun passed.

Is there a way to tell cucumber or jenkins to ignore the first test run, and only mark the rerun tests as a pass or fail?

Or is there a neater way around this?

Thanks

Answer 1

The jenkins script is pretty simple. The build fails when a script fails, that is, has non zero exit code.

Also, if you are using the junit plugin to publish your results, make sure the plugin is not failing your build when it finds a failed test in the results.

However, try to find the reason why test fail. Always get the application log for your test runs and check what happens there. If an automated test is not reliable, use manual testing.

Also, you can use cucumber hooks to take a screenshot whenever a test fails. That's a big help when you are trying to understand failed tests.

Answer 2

Sorry, if I am ignorant or just don't understand the philosophy of cucumber, but failing tests usually have a reason. In general, find out why your tests fail and then fix that issue (or your tests).

To answer you question more directly, you can check into the Log Parser Plugin. My gut feeling tells me that Jenkins correctly marks jobs as failed (or better as unstable) when a testcase fails.

Answer 3

why you don't try to get your test results using groovy

def getResults(){
AbstractTestResultAction testResultAction = currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
               def failCount = null
               def failureDiffString = null
               def totalCount = null
               def passed = null
               def passrate = null
               if (testResultAction != null) {
                   failCount = testResultAction.failCount
                   totalCount = testResultAction.totalCount
                   passed = totalCount - failCount
                   passrate = (passed/totalCount*100).toInteger()
                  }

           return passrate
}

and you can use something like this :

if(getResults() >= 95){
    currentBuild.result="SUCCESS"
    } else {
        currentBuild.result="FAILED"
        throw new Exception("Pass rate lower than 95%")
    }