Support for Hadoop 1.0.1 jobs on Amazon EMR

Question

I have developed some MR jobs using java and hadoop 1.0.1. However, EMR supports only upto Hadoop 0.20. Is it possible to run Hadoop 1.0.1 jobs on EMR or do I have to downgrade my library stack to comply with EMR hadoop version ?

Answer 1

Depends on whether you're using any 1.0.1 specific classes or not. The core Mapper and Reducer classes (both new and old API types) haven't changed between 0.20 and 1.0.1.

You can try and change your hadoop dependency to 0.20.2 and rebuild your MR job jar - if there are not compile errors then you're pretty close (there may be some bug fixes between 0.20 and 1.0.1 but i imagine you'll be ok).

If you do find that your job fails to compile, and it relates to some Input / Output formats not being available in 0.20 (like some of the Multi Input / Outputs), the you can check the Hadoop source for 1.0.1 (or indeed the Cloudera 0.20.2 source) to see if you can 'backport' the missing formats and add then into your job jar.

Feel free to re-post an compilation errors back into your original questions for people to comment on potential work arounds.