Can surefire and failsafe plugin deliberately randomize execution order of test classes?

Question

It appears that surefire and failsafe plugins execute test classes in order while tests defined within a class execute in undetermined order.

To discover tests which rely on order (what we consider bad tests) we want to force the order to be different for each run. Ideally, we'd have a mechanism to disable randomization or a seed number that would repeat the order (must like the old palm OS emulator had a seed number that drove a sequence of random tests).

Let me know if you know a way to do this? If not, I guess I can work one into a local fork and then submit it.

Thanks

Peter

Answer 1

Some of the other answers link to the surefire maven documentation page, but like most maven documentation it provides no examples of how to actually specify the settings in the maven XML morass. Here is how to do it with the surefire plugin:

<properties>
  <surefire.plugin.version>2.16</surefire.plugin.version>
</properties>

<build>
 <plugins>
   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <version>${surefire.plugin.version}</version>
     <configuration>
       <runOrder>random</runOrder>
     </configuration>
   </plugin>
 </plugins>
</build>

Answer 2

Specify the runOrder of Surefire to "random" http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder

Answer 3

I think this is more the responsibility of your unit test framework, not the Surefire/Failsafe plugins, which are just responsible for bootstrapping the test framework.

There is already a Stackoverflow question about how to make Junit tests run in random order (the answer is to use a custom ClassRunner):

How can I make my JUnit tests run in random order?

This library supplies an implementation if you don't want to write your own: http://randomjunit.sourceforge.net/

Answer 4

First it seemed to me that you are mixing things. Maven-Surefire-PLugin is responsible to run unit tests where it is the case to be independant of the order of execution. Maven-Failsafe-plugin is responsible for exectution of integration tests which is different, cause integration tests could be dependant on the order which os no problem. Apart from that maven-surefire-plugin has some possibilities to influcence of the order of execution order:

http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder

This will be of course influenced by the testing framework which you use. In JUnit you can influence the order only in limited way. In TestNG it is a complete different story, cause TestNG has the ability to define dependencies etc.

Maven-Failsafe-Plugin has the same capabilities to influence the order of executions.

http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#runOrder