Can TestNG see my Spock (JUnit) test results?

Question

Need to get TestNG to run my Spock Tests as TestNG is used throughout the rest of the system.
Since TestNG supports running JUnit tests, I tried this:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="suiteTestName" verbose="1" junit="true">
    <test name="myTestName">
        <classes>
            <class name="mypackage.artifact.SomeArtifactSpecification" />
        </classes>
    </test>
</suite>

So this actually ran the Groovy 'SomeArtifactSpecification' Spock Unit Test perfectly. I could see some I/O from the tests, but since I ran from within Eclipse I used the debugger to verify that all the tests were executing properly. My classpath has TestNG-6.8.7 and JUnit-4.10 on it. The problem is that TestNG doesn't seem to know that it did! Here is the end output:

===============================================
myTestName
Total tests run: 0, Failures: 0, Skips: 0
===============================================

Does anyone know if it is possible to get TestNG to recognize these tests?

Additional Details - added for the Bounty

In one of the comments below, Peter states that it depends upon what TestNG is doing. I investigated that as much as I could. It seems like both Junit and TestNG invoke the 'run' method of Spock's Sputnik class line 63. However, when invoked directly from JUnit, Sputnik is called from JUnit4TestClassReference. When called from TestNG, Sputnik is invoked from JUnitCore.

So the question remains: How to get TestNG to recognize the test outputs from Spock? If I am to live long and prosper, I really need to know!!! It seems like the answer may be to get TestNG to run with JUnit4TestClassReference, or to use some type of clever wrapper to make TestNG do that...

For the gory details, I posted two stack traces below. The first is from the TestNG run, which does not work, the second is from running Spock directly without TestNG.

First Stack Trace from TestNG:

org.testng.remote.RemoteTestNG at localhost:54288   
Thread [main] (Suspended (breakpoint at line 65 in ArtifactManagerSpecification))   
    ArtifactManagerSpecification.$spock_feature_0_0() line: 65  
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).invokeFeatureMethod() line: 285 
    ParameterizedSpecRunner(BaseSpecRunner).doRunIteration() line: 256  
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runIteration(Object[], int) line: 223   
    ParameterizedSpecRunner(BaseSpecRunner).initializeAndRunIteration(Object[], int) line: 214  
    ParameterizedSpecRunner(BaseSpecRunner).runSimpleFeature() line: 205    
    ParameterizedSpecRunner(BaseSpecRunner).doRunFeature() line: 199    
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 175  
    ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 152 
    ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 112   
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
    Method.invoke(Object, Object...) line: 601  
    ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
    ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
    ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
    ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 91  
    ParameterizedSpecRunner(BaseSpecRunner).run() line: 82  
    Sputnik.run(RunNotifier) line: 63   
    JUnitCore.run(Runner) line: 157 
    JUnitCore.run(Request) line: 136    
    JUnit4TestRunner.start(Class, String...) line: 81   
    JUnit4TestRunner.run(Class, String...) line: 69 
    TestRunner$1.run() line: 682    
    TestRunner.runWorkers(List<IWorker<ITestNGMethod>>, String, ListMultiMap<Integer,TestMethodWorker>) line: 1005  
    TestRunner.privateRunJUnit(XmlTest) line: 713   
    TestRunner.run() line: 614  
    SuiteRunner.runTest(TestRunner) line: 334   
    SuiteRunner.runSequentially() line: 329 
    SuiteRunner.privateRun() line: 291  
    SuiteRunner.run() line: 240 
    SuiteRunnerWorker.runSuite(SuiteRunnerMap, XmlSuite) line: 52   
    SuiteRunnerWorker.run() line: 86    
    RemoteTestNG(TestNG).runSuitesSequentially(XmlSuite, SuiteRunnerMap, int, String) line: 1224    
    RemoteTestNG(TestNG).runSuitesLocally() line: 1149  
    RemoteTestNG(TestNG).run() line: 1057   
    RemoteTestNG.run() line: 111    
    RemoteTestNG.initAndRun(String[], CommandLineArgs, RemoteArgs) line: 204    
    RemoteTestNG.main(String[]) line: 175   

Second Stack Trace (that works) running Spock directly

Thread [main] (Suspended (breakpoint at line 65 in ArtifactManagerSpecification))   
ArtifactManagerSpecification.$spock_feature_0_0() line: 65  
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).invokeFeatureMethod() line: 285 
ParameterizedSpecRunner(BaseSpecRunner).doRunIteration() line: 256  
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runIteration(Object[], int) line: 223   
ParameterizedSpecRunner(BaseSpecRunner).initializeAndRunIteration(Object[], int) line: 214  
ParameterizedSpecRunner(BaseSpecRunner).runSimpleFeature() line: 205    
ParameterizedSpecRunner(BaseSpecRunner).doRunFeature() line: 199    
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 175  
ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 152 
ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 112   
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 601  
ReflectionUtil.invokeMethod(Object, Method, Object...) line: 138    
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 330  
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 311 
ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 91  
ParameterizedSpecRunner(BaseSpecRunner).run() line: 82  
Sputnik.run(RunNotifier) line: 63   
JUnit4TestClassReference(JUnit4TestReference).run(TestExecution) line: 50   
TestExecution.run(ITestReference[]) line: 38    
RemoteTestRunner.runTests(String[], String, TestExecution) line: 467    
RemoteTestRunner.runTests(TestExecution) line: 683  
RemoteTestRunner.run() line: 390    
RemoteTestRunner.main(String[]) line: 197   

Answer 1

I investigated this further and at least I found the cause.

In org.testng.junit.JUnit4TestMethod below, Spock test names are unable to be properly resolved. I am guessing, but I think Spock uses a type of name mangling to support the nice text method name it allows. Thus, even though the TestNG code below gets a method name like "This is my test", it can not resolve that via the means shown. I believe the real method name is something like: $spock_feature_0_0

private static Method getMethod(Description desc) {
    Class<?> c = desc.getTestClass();
    String method = desc.getMethodName();
    try {
        return c.getMethod(method);
    } catch (Throwable t) {
        Utils.log("JUnit4TestMethod", 2,
                "Method '" + method + "' not found in class '" + c.getName() + "': " + t.getMessage());
        return null;
    }
}

As a result, this throws an exception on the call c.getMethod(method) - it can't resolve the Spock method name to an actual method on the class.

What seems to really 'muck up the situation' is that the code above is called by Junit (org.junit.runner.notification.RunNotifier) in this code:

private abstract class SafeNotifier {
    void run() {
        synchronized (fListeners) {
            for (Iterator<RunListener> all= fListeners.iterator(); all.hasNext();)
                try {
                    notifyListener(all.next());
                } catch (Exception e) {
                    all.remove(); // Remove the offending listener first to avoid an infinite loop
                    fireTestFailure(new Failure(Description.TEST_MECHANISM, e));
                }
        }
    }

it would be really nice for this code to not quietly remove the listener, but rather log it...