how to submit many mapreduce jobs in one terminal?
-
27-05-2021 - |
Question
I am using mapreduce to run the program, to submit one job: I can use following format ./hadoop jar program.jar arguments
in one terminal, this can submit one job, but what if I want to submit 100 jobs? impossible to
open 100 terminals
so I am seeking a batch job submitting methods, thanks!
Solution
You call ToolRunner.run(..) in a loop within your main method. Be sure to amend your Tool instance to call Job.submit()
rather than Job.waitForCompletion()
- ensuring your jobs run more in parallel rather than sequentially (you're still limited to how many jobs you can run in parallel by your cluster size / config):
public class MyDriver extends Configured implements Tool {
public static void main(String args[]) {
for (int x = 0; x < 100; x++) {
ToolRunner.run(new MyDriver(), args);
}
}
public int run(String args) {
Job job = new Job(getConf());
// job set up
// ...
job.submit();
}
}
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow