How to Report Results to Sauce Labs using Geb/Spock?

Question

I want to use the Sauce Labs Java REST API to send Pass/Fail status back to the Sauce Labs dashboard. I am using Geb+Spock, and my Gradle build creates a test results directory where results are output in XML. My problem is that the results XML file doesn't seem to be generated until after the Spock specification's cleanupSpec() exits. This causes my code to report the results of the previous test run, rather than the current one. Clearly not what I want!

Is there some way to get to the results from within cleanupSpec() without relying on the XML? Or a way to get the results to file earlier? Or some alternative that will be much better than either of those?

Some code:

In build.gradle, I specify the testResultsDir. This is where the XML file is written after the Spock specifications exit:

drivers.each { driver ->
    task "${driver}Test"(type: Test) {
        cleanTest
        systemProperty "geb.env", driver    
        testResultsDir = file("$buildDir/test-results/${driver}")
        systemProperty "proj.test.resultsDir", testResultsDir
    }
}

Here is the setupSpec() and cleanupSpec() in my LoginSpec class:

class LoginSpec extends GebSpec {
    @Shared def SauceREST client = new SauceREST("redactedName", "redactedKey")
    @Shared def sauceJobID
    @Shared def allSpecsPass = true

    def setupSpec() {
        sauceJobID = driver.getSessionId().toString()
    }

    def cleanupSpec() {
        def String specResultsDir = System.getProperty("proj.test.resultsDir") ?: "./build/test-results"
        def String specResultsFile = this.getClass().getName()
        def String specResultsXML = "${specResultsDir}/TEST-${specResultsFile}.xml"
        def testsuiteResults = new XmlSlurper().parse( new File( specResultsXML ))

        // read error and failure counts from the XML       
        def errors = testsuiteResults.@errors.text()?.toInteger()
        def failures = testsuiteResults.@failures.text()?.toInteger()

        if ( (errors + failures) > 0 ) { allSpecsPass = false }

        if ( allSpecsPass ) {
            client.jobPassed(sauceJobID)
        } else {
            client.jobFailed(sauceJobID)
        }
    }
}

The rest of this class contains login specifications that do not interact with SauceLabs. When I read the XML, it turns out that it was written at the end of the previous LoginSpec run. I need a way to get to the values of the current run.

Thanks!

Answer 1

Test reports are generated after a Specification has finished execution and the generation is performed by the build system, so in your case by Gradle. Spock has no knowledge of that so you are unable to get that information from within the test.

You can on the other hand quite easily get that information from Gradle. Test task has two methods that might be of interest to you here: addTestListener() and afterSuite(). It seems that the cleaner solution here would be to use the first method, implement a test listener and put your logic in afterSuite() of the listener (and not the task configuration). You would probably need to put that listener implementation in buildSrc as it looks like you have a dependency on SauceREST and you would need to build and compile your listener class before being able to use it as an argument to addTestListener() in build.gradle of your project.

Answer 2

Following on from erdi's suggestion, I've created a Sauce Gradle helper library, which provides a Test Listener that parses the test XML output and invokes the Sauce REST API to set the pass/fail status.

The library can be included by adding the following to your build.gradle file:

import com.saucelabs.gradle.SauceListener

buildscript {
  repositories {
    mavenCentral()
    maven {
        url "https://repository-saucelabs.forge.cloudbees.com/release"
    }
  }
  dependencies {
    classpath group: 'com.saucelabs', name: 'saucerest', version: '1.0.2'
    classpath group: 'com.saucelabs', name: 'sauce_java_common', version: '1.0.14'
    classpath group: 'com.saucelabs.gradle', name: 'sauce-gradle-plugin', version: '0.0.1'
  }
}

gradle.addListener(new SauceListener("YOUR_SAUCE_USERNAME", "YOUR_SAUCE_ACCESS_KEY"))

You will also need to output the Selenium session id for each test, so that the SauceListener can associate the Sauce Job with the pass/fail status. To do this, include the following output in the stdout:

SauceOnDemandSessionID=SELENIUM_SESSION_ID