Junit + Surefire not running the same number of test cases

Question

I have maven's surefire plugin installed for junit. I have some test suites set up with the following code(stripped to protect the innocent):

package com.company.qa.guiauto.test.suites;

import com.company.qa.guiauto.test.regression.contentcenter.TestClass1;
import com.company.qa.guiauto.test.regression.contentcenter.TestClass2;
import com.company.qa.guiauto.test.regression.contentcenter.TestClass3;
import com.company.qa.guiauto.test.regression.contentcenter.TestClass4;
import org.junit.runner.RunWith;


import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
        TestClass1.class, //20 Junit @Tests
        TestClass2.class, //33 Junit @Tests
        TestClass3.class, //4 Junit @Tests
        TestClass4.class  //13 Junit @Tests
})

public class SomeSuite {
}

However, when jenkins runs these tests the results are quite variable. Sometimes it runs 70 tests. Sometimes 68, sometimes 30. Can anyone suggest a reason why this would be variably quitting the test run before its actually done? As you can see here, we have a high of 72 tests, a low of 44 tests, and a last run of 67 tests. The number of tests has not changed.

Answer 1

I was able to find the issue.

It was an issue that was fixed in the latest version of Surefire(2.17) in ticket http://jira.codehaus.org/browse/SUREFIRE-1055.

Upgrading our version of Surefire fixed it.

Answer 2

Looking at your gist, I can see different test classes being executed in the 2 sessions:

Succesful Output File from Jenkins
##Succesful run here.  
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.company.qa.guiauto.test.regression.contentcenter.TestClass1

Unsuccessful output file
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.company.qa.guiauto.test.regression.test.TestClass1