Hadoop configuration - are mapper/combiner affected by io.sort.factor and io.sort.mb?
-
15-06-2021 - |
Pregunta
If I modify the io.sort.factor and io.sort.mb, does the local sort taking place on the map side use these variables, or are they only used by the sort done on the reducer side?
Solución
Yes, they are used on the map side too (irrespective of whether you have a combiner or not):
- MapTask.java
io.sort.factor
- line 1695io.sort.mb
- lines 932 - 944
Licenciado bajo: CC-BY-SA con atribución
No afiliado a StackOverflow