Why does Specs2 run these `sequential` tests in random order?

Question

There's an old database test suite that I'm trying to migrate from Specs to Specs2. However, Specs2 runs the tests in a weird order (from my point of view), which breaks the tests since they change the database state, and run certain code twice.

Find below a simplified version of the tests. As far as I've understood, the tests should run in this order: (since I've specified sequential):
! 222, ! 333, ! 444
However, what actually happens, is that they're executed in this order:
! 333, ! 222, ! 444

Here are the tests:

object IncludedSpec extends Specification {
  sequential
  println("sssstart")

  "IncludedSpec" should {

    println("..222")
    "context free block" >> {
      println("....! 222 the first test")
      1 === 1
    }

    var variableN = 0

    "block with context" >> {
      println("....333")
      "using one variable" >> {
        println("......! 333 doing some tests and setting variableN")
        variableN = 123
      }

      println("....444")
      "using it again" >> {
        println("......! 444 testing variableN")
        variableN === 123
      }
      println("....555")
    }
  }

  println("eeeend")
}

Here is all println output:

sssstart
eeeend
sssstart
eeeend
..222
....333
......! 333 doing some tests and setting variableN
....444
....555
....! 222 the first test
......! 444 testing variableN

And my two questions:

1. Why isn't ! 222 executed first?

2. How is it possible that sssstart eeeend is output twice? The spec is an object and isn't created twice?

Oddly enough, if I remove the side effects from the tests — that is, removing variableN and replacing the test bodies with ok — the tests run in the correct order.

Version details: I'm running these tests with Paly Framework 2.1-SNAPSHOT (version 203df0e from October 28 2012) and Scala 2.10.0-RC1. I think the Specs2 version bundled with Play is version 1.12, because the inline method is available, and it was added in 1.12(-SNAPSHOT), see https://github.com/etorreborre/specs2/issues/87 and there are no later Specs2 versions.

(Oh, and if you think I should completely rewrite the tests, then have a look at this question instead: How design a Specs2 database test, with interdependent tests? )

Answer 1

Originally, in specs2, you could create the following:

1 - an example with in: "this is an example" in { 1 must_== 1 }

2 - an example with >>: "this is an example" >> { 1 must_== 1 }

3 - a block of examples with >>

"this system should" >> {
  "do something" >> ok
  "do something else" >> ok
}

The important thing was that in was reserved for examples alone and accepted anything that could be converted to a Result. On the other hand >> could be used for both examples and groups of examples (to have a uniform style of nesting) so it accepted values of type Example or Result.

Now things started to become a bit more complicated when you wanted to do the following:

1 - use foreach to create a group of examples

"this group has 5 examples" in {
  (1 to 5) foreach { i => "example "+i >> ok }
}

2 - use foreach to create a group of expectations

"this example has 5 expectations" in {
  (1 to 5) foreach { i => i must_== i }
}

The trouble is, those 2 expressions with foreach both have type Unit. But they're doing 2 very different things! The first one is building examples, so this expression needs to be evaluated right away to build the Specification. The second one is creating the body of an Example and will be executed later (or maybe never if the example is filtered out for example). Two things, with the same operator >>, that cannot work.

So it was decided that >>(block: =>Unit) means "this builds a group of examples by a side-effect", and in(expectations: =>Unit) means "this builds the body of an Example probably having expectations which will be a side-effect.

Now, when this explains a bit better why you're seeing a strange order with your print statements:

..222
....333
......! 333 doing some tests and setting variableN
....444
....555

are printed first because they are contained in blocks of type Unit, interpreted as groups of Examples.

And :

....! 222 the first test
......! 444 testing variableN

are printed consequently because they are in blocks of type MatchResult[_], that is, they are considered as the body of examples.

I agree that this is confusing and I hope that this explanation brings some perspective on why things are the way they are. Of course another lesson is "side-effects are sneaky because they don't tell you what they are really doing".

So the general specs2 tip is to always end your blocks with proper values (unless you use a foreach construct as shown in my example above). For example, adding ok at the end of the blocks where you do a variable assignment should solve your issue.

Answer 2

(Update: This didn't work with my real rather huge Spec. It's still being executed in seemingly random order, Specs2 seems to start somewhere in the middle. — However when I did the below-mentioned replacement on a smaller Spec with perhaps 10 or 20 examples, it really did result in the spec being executed in the correct order. Weird)

Here's a partial workaround, but not really an answer:

Use in instead of >>, before the test fragments. Then the tests are executed in the correct order. (But sssstart eeeend still happens twice.)

So this works as intended: (I've replaced certain >> with in)

object IncludedSpec extends Specification {
  sequential
  println("sssstart")

  "IncludedSpec" should {

    println("..222")
    "context free block" in {   // not >>
      println("....! 222 the first test")
      1 === 1
    }

    var variableN = 0

    "block with context" >> {
      println("....333")
      "using one variable" in {   // not >>
        println("......! 333 doing some tests and setting variableN")
        variableN = 123
      }

      println("....444")
      "using it again" in {   // not >>
        println("......! 444 testing variableN")
        variableN === 123
      }
      println("....555")
    }
  }

  println("eeeend")
}

---

My confused thoughts below:

1.

I think that a difference between in and >> is that:
in can only appear before test fragments (that is, a {... x must_== y ... } block). But
>> can appear both before whole examples (with nested test fragments) and before test fragments.

So when I'm writing >>, Specs2 doesn't know directly what to expect. I don't know why this would change the test execution order though. Unless Specs2 needs to ... execute some code inside the { ...} block to find out? But doesn't Specs2 use reflection? This feels weird.

2.

If the tests have no side effects at all (in this case, update no shared vars). Or if all tests have side effects. Then they are, as far as I can tell, executed in the correct order. However, if some tests have no side effects, except for reading a shared var, then they seem to be executed after tests that write to a var. (If you use >> instead of in.) This also feels weird. Anyway I suppose that soon I'll start porting my tests to Specs2 and replacing >> with in :-)