Maven + Surefire return code is 0 on failed tests

Question

I have a project with tests split in unit and integration phases. I have it running buildbot and the problem is that even in tests are failing maven return code is 0 so buildbot build is succesful.

This is the result of mvn integration-test:

Results :

Tests in error: 
 Info about failed tests

Tests run: 5, Failures: 0, Errors: 5, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 19 seconds
[INFO] Finished at: Tue Feb 12 09:43:53 UTC 2013
[INFO] Final Memory: 36M/97M
[INFO] ------------------------------------------------------------------------

$ echo $?
0

Result for mvn install is the same without the build successful part Results :

Tests in error: 
  Info about failed tests

Tests run: 5, Failures: 0, Errors: 5, Skipped: 0

$ echo $?
0

Surefire configuration is like this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.13</version>
  <configuration>
    <printSummary>true</printSummary>
    <excludedGroups>com.testlib.IntegrationTest</excludedGroups>
  </configuration>
  <executions>
    <execution>
    <id>unit-tests</id>
    <phase>test</phase>
    <goals>
      <goal>test</goal>
    </goals>
    <configuration>
          <excludedGroups>com.testlib.IntegrationTest</excludedGroups>
    </configuration>
    </execution>
    <execution>
    <id>integration-tests</id>
    <phase>integration-test</phase>
    <goals>
      <goal>test</goal>
    </goals>
    <configuration>
      <includes>
            <groups>com.testlib.IntegrationTest</groups>
      </includes>
        </configuration>
    </execution>
  </executions>
</plugin>

I've reading other threads about maven return codes but in theory related bugs should be fixed in my maven version (Apache Maven 2.2.1 (rdebian-8))

Is there any way to change this behaviour?

Update: As suggested I tried with surefire:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.13</version>
    <dependencies>
        <dependency>
        <groupId>org.apache.maven.surefire</groupId>
        <artifactId>surefire-junit47</artifactId>
        <version>2.13</version>
        </dependency>
    </dependencies>
    <configuration>
        <groups>com.testlib.IntegrationTest</groups>
    </configuration>
    <executions>
        <execution>
            <goals>
            <goal>integration-test</goal>
            <goal>verify</goal>
            </goals>
            <configuration>
                <includes>
                  <include>**/*.class</include>
                </includes>
            </configuration>
        </execution>
    </executions>

I need the surefire-junit to avoid initialization errors.

Answer 1

I managed to have it working with two different configurations, using groups and using folders using just surefire

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.13</version>
    <configuration><excludedGroups>com.testlib.IntegrationTest</excludedGroups>
    </configuration>
    <executions>
        <execution>
            <id>integration-test</id>
            <phase>integration-test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
              <skip>false</skip>
                      <excludedGroups>com.biicode.testlib.UnitTest</excludedGroups>
                      <groups>com.testlib.IntegrationTest</groups>
            </configuration>
        </execution>
    </executions>

 <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <excludes>
            <exclude>**/integration/*.java</exclude>
        </excludes>
    </configuration>
    <executions>
    <execution>
        <id>integration-test</id>
        <phase>integration-test</phase>
        <goals>
            <goal>test</goal>
        </goals>
        <configuration>
        <skip>false</skip>
        <excludes>
            <exclude>none</exclude>
        </excludes>
        <includes>
            <include>**/integration/*.java</include>
        </includes>
    </configuration>
  </execution>
  </executions>
</plugin>

Answer 2

First check if there is same parent pom which configures:

<testFailureIgnore>true</testFailureIgnore>

somewhere...you can check this via:

mvn help:effective-pom

Furthermore you are trying to run integration test with maven-surefire-plugin which is simply wrong. For integration tests use the maven-failsafe-plugin. One other thing is to name your integration tests in the right way like IT*.java, *IT.java etc.

One other thing is why are you using such an old Maven version check Maven 3.0.4.

Ah sorry. Oversight that you are talking about integration tests. If you correctly using the maven-failsafe-plugin for integration tests it contains a particular goal verify which is intended to check the result of the integration tests afterwards. But you need to separately configure that via an execution block and binding to a particular lifecycle phase.