Maven build [WARNING] we have a duplicate class

Question

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/NoOpLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/Jdk14Logger.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar

I've looked into my local m2 repo, I have two classes there in commons-logging-api jar, LogFactoryImpl.class and LogFactoryImpl$1.class. Same as all the classes mentioned in the warnings.

One thing to mention is that I am using shade plugin in my pom.xml.

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.4</version>
            <configuration>
                <createDependencyReducedPom>true</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.~~~~black out my own main class here~~~~~</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>

I noticed that the dependency tree looks like as below

[INFO] +- org.apache.cxf:cxf-bundle-jaxrs:jar:2.5.1:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] \- org.apache.hadoop.hive:hive-jdbc:jar:0.7.1-cdh3u3:compile
[INFO]    \- org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile
[INFO]       \- commons-logging:commons-logging-api:jar:1.0.4:compile

and commons-logging.jar and commons-logging-api.jar both have org/apache/commons/logging/LogFactory.class.

somehow Shad plugin is trying to squeeze them in to a big fat jar at the end. then the warning is showing up. It's been said this is ignorable warning. But I am a bit worried, How does the application know what is the exact class should be used if there are two duplicated class with the same name?

Answer 1

Take a look at the "Dependency Exclusions" section in the Maven doc.

In your provided example, I'll exclude the commons-logging:commons-logging-api:jar:1.0.4:compile dependency from org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile. In your pom.xml :

    <dependency>
        <groupId>org.apache.hadoop.hive</groupId>
        <artifactId>hive-common:jar</artifactId>
        <version>0.7.1-cdh3u3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Answer 2

You might also have run into a limitation of maven-shader-plugin. It replaces the default jar artifact (created by maven-jar-plugin). This works fine on a clean build, but on a rebuild where the jar is not regenerated, the shader runs again on the jar it created last time, which already contains copies of all the class dependencies. That produces a lot of warnings about duplicates.

This issue is still unaddressed as of maven-shader-plugin 2.0: https://issues.apache.org/jira/browse/MSHADE-126

One workaround is to add the maven-jar-plugin explicitly to your pom.xml and add the configuration setting <forceCreation>true</forceCreation>.

Answer 3

In my case, my parent pom was including commons-beanutils and my child module (which is the only thing I wanted to compile) was including commons-io.

The shade plug in complained about duplicates since commons-io and commons-beansutil shared some common classes. Note that beansutiul was being included even though it was not needed, and was not used.

I solve this by minimizing the jar by adding this to the configuration:

<minimizeJar>true</minimizeJar>

Now the shade plugin did not add unused resources.

Warning went away.

Answer 4

All above (about reviewing dependencies tree and excluding) is correct in the most of cases, but in my case (i didn't have overlapping in my dependencies) preliminary clean helped (don't know why though):

mvn clean package

Answer 5

You can exclude the jar you don't want (the ones that are giving the duplicate warnings using the following tags under the shade plugin -

    <configuration>
    <artifactSet>
      <excludes>
        <exclude>commons-logging:commons-logging</exclude>
      </excludes>
    </artifactSet>
    <minimizeJar>true</minimizeJar>
    </configuration>

More details can be found at http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html

Answer 6

I saw this happen in eclipse when I updated my parent project's dependencies.

I deleted all the files in my target directory and it fixed the issues.

Answer 7

In my case I was relying on a package that also creates a shaded jar.

Shaded jars are meant for deployment, not installing as a dependency.

Creating a reduced dependency POM during the build process of the dependency, instructs maven on which dependencies can be left out.

In the maven-shade-plugin configuration:

<configuration>
  <createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>

For more details see this post:

What is the maven-shade-plugin used for, and why would you want to relocate java packages?

The error I was getting from maven:

WARNING: x.jar, y.jar contain overlapping classes

Answer 8

You have dependencies in your pom which contain duplicate classes but without the appropriate pom i couldn't say a word about it.