Sqoop 1.4.4 in CDH4.3 incompatibleClassChangeError

Question

Sqoop 1.4.4 supports importing from db to HBase with composite row key while before 1.4.4, only one column from the db can be used as the row key. So far, CDH4.3 and HDP1.3 both only supports Sqoop 1.4.3. I tried to swap in sqoop 1.4.4 to my CDH4.3 environment. When I run a simple Sqoop job, I've Got the error below:

13/08/12 23:36:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201308122236_0001
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
    at org.apache.sqoop.mapreduce.DelegatingOutputFormat.checkOutputSpecs(DelegatingOutputFormat.java:63)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
    at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
    at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
    at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
    at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
    at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

Anybody got this before? Can someone give me a pointer Sqoop 1.4.4 is compatible with what versions of mapreduce,HBase and HDFS?

Answer 1

Hadoop has gone through a huge code refactoring from Hadoop 1.0 to Hadoop 2.0 (correspondingly from CDH3 to CDH4). One side effect is that code compiled against Hadoop 1.0 (CDH3) is not compatible with Hadoop 2.0 (CDH4) and vice-versa. However source code is compatible and thus one just need to recompile code with target Hadoop distribution.

The exception "Found class X, but interface was expected at" is very common when you're running code that is compiled for Hadoop 1.0 (CDH3) on Hadoop 2.0 (CDH4) or vice-versa.

Solution is simple, you need to synchronise the versions. You have to make sure that you're using binary artifact compiled for your Hadoop distribution. Sqoop makes this easy as the target hadoop distribution is encoded in the artifact name - for example sqoop-1.4.4.bin__hadoop-2.0.4-alpha.tar.gz is meant to be used on Hadoop 2.0 and CDH4 [1].

Jarcec

Links:

1: http://www.apache.org/dist/sqoop/1.4.4/