Using camel-hdfs component with Hadoop 2

Question

I am fairly new to servicemix and osgi, I was wondering if anyone has had any success using the camel hdfs component with a hadoop 2.x cluster in servicemix.

I attempted this but got an IPC version misatch issue. To get arround it, I forked the camel-hdfs component, modified the hadoop dependencies and with very little change to the component got it working when deployed as a standalone application with all the tests passing.

The challenge now is running it in servicemix, there's issues with initializing hadoop classes even though I'm wrapping the dependent hadoop libraries as a bundle using the wrap:* command. An example exception I get is

Exception in thread "SpringOsgiExtenderThread-78" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/HdfsBlockLocation
at      
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
.....
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.HdfsBlockLocation not found by wrap_mvn_org.apache.hadoop_hadoop-common_2.0.0-cdh4.2.0 [441]
at
org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:812)

Answer 1

camel-hdfs is supporting Hadoop 1.1 or 1.2 (depending on Camel version in use). This component has not been upgraded to support Hadoop 2.

And when you run in ServiceMix then you would need OSGi bundles for Hadoop 2. Installing them as wrap isn't always possible.

So answer is, that Hadoop is not yet supported with camel-hdfs.