hadoop
and spark
version
s should be in sync. (In my case, I am working with spark-1.2.0
and hadoop 2.2.0
)
STEP 1 - goto $SPARK_HOME
STEP 2 - Simply mvn build
spark with the version of hadoop
client you want,
mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean package
STEP 3 - Also spark project should have proper spark version,
name := "smartad-spark-songplaycount"
version := "1.0"
scalaVersion := "2.10.4"
//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.2.0"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"