How to increase the number of containers in nodemanager in YARN
Question
A node in my YARN cluster has 64GB memory and 24 cores. I set the following properties in the yarn-site.xml:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>32768</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>16</value>
</property>
But I found still the nodemanager in the node has only 7 containers. What are other properties I need to set?
Solution
You need to tell YARN how to break down the memory to containers so for instance if you set the memory per container to 2GB will give you 16 containers
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
OTHER TIPS
try something like:
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
</property>
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow