assembly-merge-strategy issues using sbt-assembly

Question 1

I think it should be MergeStrategy.first with a capital M, so mergeStrategy in assembly := MergeStrategy.first.

Question 2

As for the current version 0.11.2 (2014-03-25), the way to define the merge strategy is different.

This is documented here, the relevant part is:

NOTE: mergeStrategy in assembly expects a function, you can't do
mergeStrategy in assembly := MergeStrategy.first

The new way is (copied from the same source):

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
    case "application.conf" => MergeStrategy.concat
    case "unwanted.txt"     => MergeStrategy.discard
    case x => old(x)
  }
}

This is possibly applicable to earlier versions as well, I don't know exactly when it has changed.

Question 3

this is the proper way to merge most of the common java/scala projects. it takes care of META-INF and classes.

also the service registration in META-INF is taken care of.

assemblyMergeStrategy in assembly := {
case x if Assembly.isConfigFile(x) =>
  MergeStrategy.concat
case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
  MergeStrategy.rename
case PathList("META-INF", xs @ _*) =>
  (xs map {_.toLowerCase}) match {
    case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
      MergeStrategy.discard
    case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
      MergeStrategy.discard
    case "plexus" :: xs =>
      MergeStrategy.discard
    case "services" :: xs =>
      MergeStrategy.filterDistinctLines
    case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
      MergeStrategy.filterDistinctLines
    case _ => MergeStrategy.first
  }
case _ => MergeStrategy.first}

Question 4

I have just setup a little sbt project that needs to rewire some mergeStrategies, and found the answer a little outdated, let me add my working code for versions (as of 4-7-2015)

sbt 0.13.8
scala 2.11.6

assembly 0.13.0

mergeStrategy in assembly := {
  case x if x.startsWith("META-INF") => MergeStrategy.discard // Bumf
  case x if x.endsWith(".html") => MergeStrategy.discard // More bumf
  case x if x.contains("slf4j-api") => MergeStrategy.last
  case x if x.contains("org/cyberneko/html") => MergeStrategy.first
  case PathList("com", "esotericsoftware", xs@_ *) => MergeStrategy.last // For Log$Logger.class
  case x =>
     val oldStrategy = (mergeStrategy in assembly).value
     oldStrategy(x)
}

Question 5

For the new sbt version (sbt-version :0.13.11), I was getting the error for slf4j; for the time being took the easy way out : Please also check the answer here Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.class where sbt-dependency-graph tool is mentioned which is pretty cool to do this manually

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

Question 6

Quick update: mergeStrategy is deprecated. Use assemblyMergeStrategy. Apart from that, earlier responses are still solid

Question 7

Add following to build.sbt to add kafka as source or destination

 assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 //To add Kafka as source
 case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => 
 MergeStrategy.concat
 case x => MergeStrategy.first
 }