Hadoop 2.2 chain MapReduce jobs Map -> Reducer -> Map -> Reducer

Question

I have two MapReduce jobs, the output of the first Reducer is the input of the second Mapper :

Map1 -> Reduce1 -> Map2 -> Reduce2

For now Map2 reads from the files outputted by Reduce1. So Map1 -> Reduce1 and Map2 -> Reduce2 are independant.

It works, but it would be easier and I think more optimized if the output of Reduce1 was directly the input of Map2.

Is there a way to do that ? In this case Map2 would be just an identity mapper so it would be even better if I could do :

Map1 -> Reduce1 -> Reduce2

Reduce1, Map2 and Reduce2 have the same input and output types.

Thanks !

Answer 1

According to my understanding these points i can tell you (may or may not help you, correct me if i am wrong):

1) map 1-> reduce 1-> directly to mapper2 : for optimizations are addressed in spark cluster computing framework (using in-memory computations, avoiding unnecessary read/writes to hdfs).

2) if you want something like reducer1 ->reducer2 . you have to think how you can write the logic in one reducer itself , but the problem here is its all depends on your requirement i mean the aggregation on which keys you want to perform (in more detail : reducer1 receives same set of key, on which only u can act the task of next aggregation).

3) In Hadoop the protocol is like this only : map --> then aggregation , if any next aggregation , it has to come from a Userdefinedmapper/IdentityMapper.

hope this helps :)