TestNG Ant tasks vs Surefire

Question

I was wondering how different surefire is when executing TestNG than TestNG ant tasks? The reason is that I am seeing consistent difference in behavior when trying to run a TestNG test that extends a JUnit test base (this is a workaround to run JBehave tests in TestNG described here: http://jbehave.org/documentation/faq/). Surefire detects my test as a JUnit test incorrectly (probably because its base is TestCase), while the Ant tasks run perfectly. Can anyone provide an insight into how TestNG handle both cases?

The test looks as follows:

public class YourScenario extends JUnitScenario {

@org.testng.annotations.Test 
public void runScenario() throws Throwable {
    super.runScenario();
}
}

Answer 1

The short answer is that the ant task is part of the TestNG distribution, so it's part of our tests and I always make sure that it remains up to date with TestNG.

Surefire is developed as part of the Maven project, and as such, it sometimes lags behind (and just like you, I have sometimes encountered bugs when running my tests with Surefire that didn't happen when running from the command line/ant/Eclipse).

I'll bring this question to the Maven team's attention, maybe they'll have more to say.

Answer 2

This looks to be a known bug: http://jira.codehaus.org/browse/SUREFIRE-575.

Have you tried using a TestNG XML suite definition instead of Surefire's automatic test case detection?