ScalaTest: Running a test 50 times

Question

Is there a simple way to implement failure tolerance in ScalaTest? I'm looking to run the same test 50 times and give it a tolerable error margin, e.g. 10%.

In the above case, the test would only pass if 45 out of 50 tests were successful.

Answer 1

The best way to do that is to override withFixture and rerun failed tests using whatever algorithm makes sense in your particular case. For inspiration, I'd suggest you look at the Retries trait in ScalaTest itself. The Scaladoc is here:

http://doc.scalatest.org/2.1.0/index.html#org.scalatest.Retries

The actual source code for Retries is here:

https://github.com/scalatest/scalatest/blob/master/src/main/scala/org/scalatest/Retries.scala

Answer 2

Here is an addition to Bill Venners proposed solution. I needed to implement a few retries for flickering/unstable tests.


val retries = 4

override def withFixture(test: NoArgTest) = {
  if (isRetryable(test)) withFixture(test, retries) else super.withFixture(test)
}

def withFixture(test: NoArgTest, count: Int): Outcome = {
  val outcome = super.withFixture(test)
  outcome match {
    case Failed(_) | Canceled(_) => if (count == 1) super.withFixture(test) else withFixture(test, count - 1)
    case other => other
  }
}

Extended test class for retries (with Retries) and each test with taggedAs Retryable. Such tests when needed will be retried up to 4 times.

Answer 3

Scala-check may be a good solution for that. http://www.scalacheck.org/