Maven Surefire parallel='both' working the same way as 'methods'

Question

I have three test classes: FirstTest, SecondTest and ThirdTest. This is how FirstTest class looks like:

public class FirstTest {

  @Test
  public void test() throws InterruptedException {
    System.out.println(this.getClass() + " " + new Date());
    Thread.sleep(10_000);
  }

  @Test
  public void test2() throws InterruptedException {
    System.out.println(this.getClass() + " " + new Date());
    Thread.sleep(10_000);
  }

  @Test
  public void test3() throws InterruptedException {
    System.out.println(this.getClass() + " " + new Date());
    Thread.sleep(10_000);
  }

  @Test
  public void test4() throws InterruptedException {
    System.out.println(this.getClass() + " " + new Date());
    Thread.sleep(15_000);
  }

}

SecondTest class is identical (10s+10s+10s+15s = 45s total), and ThirdTest contains 10s+10s+10s test methods.

I'm using Maven Surefire plugin with following configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.9</version>
    <configuration>
      <excludes>
        <exclude>${excludeTestPath}</exclude>
      </excludes>
      <parallel>both</parallel>
      <threadCount>20</threadCount>
    </configuration>
</plugin>

I'm having problem with understanding how parallel attribute works. I've tried setting "both", "methods" and "classes" values, but got some confusing output. Please, have a look and explain it to me:

parallel='classes'

Concurrency config is parallel='classes', perCoreThreadCount=true, 
threadCount=20, useUnlimitedThreads=false
Running experiment.ThirdTest
class experiment.ThirdTest Mon Jun 10 12:00:38 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:00:48 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:00:58 CEST 2013
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.024 sec
Running experiment.SecondTest
class experiment.SecondTest Mon Jun 10 12:00:38 CEST 2013
class experiment.SecondTest Mon Jun 10 12:00:48 CEST 2013
class experiment.SecondTest Mon Jun 10 12:00:58 CEST 2013
class experiment.SecondTest Mon Jun 10 12:01:08 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.054 sec
Running experiment.FirstTest
class experiment.FirstTest Mon Jun 10 12:00:38 CEST 2013
class experiment.FirstTest Mon Jun 10 12:00:48 CEST 2013
class experiment.FirstTest Mon Jun 10 12:00:58 CEST 2013
class experiment.FirstTest Mon Jun 10 12:01:08 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.074 sec

Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

[INFO] Total time: 52.319s
[INFO] Finished at: Mon Jun 10 12:01:48 CEST 2013
[INFO] Final Memory: 22M/354M

This is expected result, three classes were executed parallel, all starting at 12:00:38. Build should take 10s+10s+10s+15s, so about 45s.

parallel='methods'

Concurrency config is parallel='methods', perCoreThreadCount=true,
threadCount=20, useUnlimitedThreads=false
Running experiment.FirstTest
class experiment.FirstTest Mon Jun 10 12:10:23 CEST 2013
class experiment.FirstTest Mon Jun 10 12:10:23 CEST 2013
class experiment.FirstTest Mon Jun 10 12:10:23 CEST 2013
class experiment.FirstTest Mon Jun 10 12:10:23 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.142 sec
Running experiment.SecondTest
class experiment.SecondTest Mon Jun 10 12:10:38 CEST 2013
class experiment.SecondTest Mon Jun 10 12:10:38 CEST 2013
class experiment.SecondTest Mon Jun 10 12:10:38 CEST 2013
class experiment.SecondTest Mon Jun 10 12:10:38 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.037 sec
Running experiment.ThirdTest
class experiment.ThirdTest Mon Jun 10 12:10:53 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:10:53 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:10:53 CEST 2013
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.009 sec

Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

[INFO] Total time: 47.405s
[INFO] Finished at: Mon Jun 10 12:11:04 CEST 2013
[INFO] Final Memory: 22M/354M

This is expected result, all methods were executed parallel within single class. Build should take 15s+15s+10s, so about 45s.

parallel='both'

Concurrency config is parallel='both', perCoreThreadCount=true, 
threadCount=20, useUnlimitedThreads=false
Running experiment.FirstTest
class experiment.FirstTest Mon Jun 10 12:18:40 CEST 2013
class experiment.FirstTest Mon Jun 10 12:18:40 CEST 2013
class experiment.FirstTest Mon Jun 10 12:18:40 CEST 2013
class experiment.FirstTest Mon Jun 10 12:18:40 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.18 sec
Running experiment.SecondTest
class experiment.SecondTest Mon Jun 10 12:18:55 CEST 2013
class experiment.SecondTest Mon Jun 10 12:18:55 CEST 2013
class experiment.SecondTest Mon Jun 10 12:18:55 CEST 2013
class experiment.SecondTest Mon Jun 10 12:18:55 CEST 2013
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.031 sec
Running experiment.ThirdTest
class experiment.ThirdTest Mon Jun 10 12:19:10 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:19:10 CEST 2013
class experiment.ThirdTest Mon Jun 10 12:19:10 CEST 2013
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.025 sec

Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

[INFO] Total time: 47.521s
[INFO] Finished at: Mon Jun 10 12:19:20 CEST 2013
[INFO] Final Memory: 22M/354M

Now, this one is confusing for me. The build took about 45s, while it should take about 15s (longest test method). Why didn't all three classes start executing at 12:18:40? Doesn't parallel='both' mean that both classes and methods are executed at the same time?

EDIT: I've found blogpost that mentions the issue.

Answer 1

See this comment from SUREFIRE-814 (from Jan. 2012):

"both" is largely untested for a long time and I have no reason to disbelieve that you're seeing this issue.

It's a safe bet that parallel=both has not received much attention in the interim. I verified that the issue affects Surefire 2.15 and I plan to comment as such and vote on the ticket (as soon as I get my signup confirmation). I recommend more people vote on the ticket.

Answer 2

This issue is fixed in 2.16. Similar issues should work properly as well https://issues.apache.org/jira/browse/SUREFIRE-797