How can I run MapReduce ToolRunner jobs with Oozie?

Question

I developed a MapReduce application which uses ToolRunner. Based on the arguments and configuration properties passed by the user the application sets up the job in a certain way: chooses a particular input/output format, chooses the mapper and reducer classes etc.. This logic is implemented in the run(String[] args) method from my main class, which implements Configured and Tool.

Oozie supports MapReduce action nodes, but the problem is that I can't use them with my custom ToolRunner application. The MapReduce action only allows to define a static job by configuring input/output formats, mapper/reducer classes and key/value classes. My application does this dynamically based on the higher level business logic.

The action should be able to execute something like this command line:

hadoop jar my.jar com.example.MyMainClass -D my.config1=a -D my.config2=b arg1 arg2

Can I create Oozie actions which support ToolRunner MapReduce applications?

Answer 1

Yes, you can use a Java Action. It is, unfortunately, not ideal because Oozie creates a job to then run your ToolRunner which creates it's own job(s). It does work though.