Submitting applications externally via REST APIs

Question

Is there currently a way to submit applications externally via the supplied REST APIs for MapReduceV1 and/or YARN? I'm hoping to find a way to do this without adding a custom service.

So far I've only figured out how to GET the application status from the ResourceManager using YARN.

Maybe I'm looking at this the wrong and there's a better way to do this externally?

Answer 1

So after doing some research, I've decided that the Oozie Workflow Scheduler is the way to go.

This is a sample workflow that can be submitted to a REST endpoint running inside your Hadoop system to start a MapReduce job. <action>s are not limited to MapReduce.

<workflow-app xmlns='uri:oozie:workflow:0.1' name='map-reduce-wf'>
    <start to='hadoop1' />
    <action name='hadoop1'>
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.mapper.class</name>
                    <value>org.apache.oozie.example.SampleMapper</value>
                </property>
                <property>
                    <name>mapred.reducer.class</name>
                    <value>org.apache.oozie.example.SampleReducer</value>
                </property>
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                <property>
                    <name>mapred.input.dir</name>
                    <value>input-data</value>
                </property>
                <property>
                    <name>mapred.output.dir</name>
                    <value>output-map-reduce</value>
                </property>
                <property>
                  <name>mapred.job.queue.name</name>
                  <value>unfunded</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name='end' />
</workflow-app>

Sample taken from https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases