Install oozie on Hadoop 2.2

Question

I need some guidance on installing Oozie on Hadoop 2.2. The Quick Start docs page indicates that

IMPORTANT: By default it builds against Hadoop 1.1.1. It's possible to build against Hadoop 2.x versions as well, but it is strongly recommend to use a Bigtop distribution if using Hadoop 2.x because the Oozie sharelibs built from the tarball distribution will not work with it.

I haven't been able to get Bigtop to work.

I tried following some guidance from here but it only tells me to edit the pom.xml files, not what to edit in them.

I have pig and maven installed.

Thanks in advance

Answer 1

This is a problem with the releases resolving shared libraries with Maven, and has been since fixed if you use git master. I had this problem so hopefully this solution will work for the Oozie version you are building from.

The advice at here is of use. Similar to the blog post you linked, the grep command will indicate the offending files:

$ grep -l "2.2.0-SNAPSHOT" `find . -name "pom.xml"`
./hadooplibs/hadoop-2/pom.xml
./hadooplibs/hadoop-distcp-2/pom.xml
./hadooplibs/hadoop-test-2/pom.xml
./pom.xml

Any mentions of 2.2.0-SNAPSHOT in these files should be replaced with 2.2.0

I would suggest removing the -SNAPSHOT part using the following command:

 $ grep -l "2.2.0-SNAPSHOT" `find . -name "pom.xml"` | xargs sed -i 's|2.2.0-SNAPSHOT|2.2.0|g'

UPDATE: If you don't have Hadoop JARs built from when you built Hadoop itself then you will need to add the option -DincludeHadoopJars

And then build the package:

 $ mvn clean package assembly:single -Dhadoop.version=2.2.0 -DskipTests

Or if you're using JDK7 and/or targeting Java 7 (as I did):

 $ mvn clean package assembly:single -Dhadoop.version=2.2.0 -DjavaVersion=1.7 -DtargetJavaVersion=1.7 -DskipTests

Documentation on building Oozie (version 4 docs) is available here.

The above worked building release-4.0.0 with Hadoop 2.2 and Java SDK 7.

The distro can then be found in distro/target.