Increasing Reducer Count above 1 on a Single Node Hadoop cluster cases serial reducer execution

Question

I'm testing some Capacity Schedule settings on a single node HDP 1.3.2 instance before pushing it to a cluster. I am aware that the default number of reducers is 1 for an MR job unless you override it, and I thought there might be some peculiarities working with a single node cluster in respect of reducers too - which might be causing this problem.

Values that I have set in the mapred-site.xml file are:

mapred.cluster.map.memory.mb=768
mapred.tasktracker.map.tasks.maximum=8
mapred.cluster.max.map.memory.mb=6144
mapred.job.map.memory.mb=1536

mapred.cluster.reduce.memory.mb=512
mapred.tasktracker.reduce.tasks.maximum=8
mapred.cluster.max.reduce.memory.mb=4096
mapred.job.reduce.memory.mb=2048

capacity-shedule.xml:

mapred.capacity-scheduler.queue.default.capacity=50 
(4 reducers appear on localhost:50030/scheduler for this default queue)
mapred.capacity-scheduler.queue.queue1.capacity=25
mapred.capacity-scheduler.queue.queue2.capacity=25

Im running a 1M row terasort program with the following arguments:

hadoop jar /usr/lib/hadoop/hadoop-examples.jar terasort -Dmapred.job.queue.name=default -Dmapred.reduce.tasks=4 ./terasort-input-small ./terasort-output

It does indeed run 4 reduce tasks... but in serial, not parallel. I then disabled the capacity-schedule.xml and ran a default 100, and... 2 achieve two reducers at a time in parallel - progress!

I then thought it must be my capacity-schedule.xml document but all the values seem to have no limit on concurrent reduce tasks either.

mapred.capacity-scheduler.default-init-accept-jobs-factor=10
mapred.capacity-scheduler.default-maximum-active-tasks-per-queue=200000
mapred.capacity-scheduler.default-maximum-active-tasks-per-user=100000
mapred.capacity-scheduler.default-minimum-user-limit-percent=100
mapred.capacity-scheduler.default-supports-priority=false
mapred.capacity-scheduler.default-user-limit-factor=1
mapred.capacity-scheduler.init-poll-interval=5000
mapred.capacity-scheduler.init-worker-threads=5
mapred.capacity-scheduler.maximum-system-jobs=3000
mapred.capacity-scheduler.queue.default.init-accept-jobs-factor=10
mapred.capacity-scheduler.queue.default.maximum-capacity=-1
mapred.capacity-scheduler.queue.default.maximum-initialized-active-tasks=200000
mapred.capacity-scheduler.queue.default.maximum-initialized-active-tasks-per-user=100000
mapred.capacity-scheduler.queue.default.minimum-user-limit-percent=100
mapred.capacity-scheduler.queue.default.supports-priority=false
mapred.capacity-scheduler.queue.default.user-limit-factor=1
mapred.capacity-scheduler.queue.default.capacity=50
mapred.capacity-scheduler.queue.queue1.capacity=25
mapred.capacity-scheduler.queue.queue2.capacity=25

Can anybody shed some light on this strange sequential execution of reducers?

Answer 1

Okay the solution was to increase mapred.cluster.reduce.memory.mb

mapred.cluster.reduce.memory.mb=768MB #allowed 2 reducers to run at once
mapred.cluster.reduce.memory.mb=1536MB #allowed all 4 reducers to run at once.

The problem was that the Slot size needed to equal the Job Task size ie

mapred.job.reduce.memory.mb = mapred.cluster.reduce.memory.mb