Hadoop configuration - are mapper/combiner affected by io.sort.factor and io.sort.mb?
-
15-06-2021 - |
Pergunta
If I modify the io.sort.factor and io.sort.mb, does the local sort taking place on the map side use these variables, or are they only used by the sort done on the reducer side?
Solução
Yes, they are used on the map side too (irrespective of whether you have a combiner or not):
- MapTask.java
io.sort.factor
- line 1695io.sort.mb
- lines 932 - 944
Licenciado em: CC-BY-SA com atribuição
Não afiliado a StackOverflow