hadoop streaming: how to see application logs?
-
11-02-2021 - |
Domanda
I can see all hadoop logs on my /usr/local/hadoop/logs
path
but where can I see application level logs? for example :
mapper.py
import logging
def main():
logging.info("starting map task now")
// -- do some task -- //
print statement
reducer.py
import logging
def main():
for line in sys.stdin:
logging.info("received input to reducer - " + line)
// -- do some task -- //
print statement
Where I can see logging.info
or related log statements of my application?
I am using Python
and using hadoop-streaming
Thank you
Soluzione
Hadoop streaming uses STDIN/STDOUT for passing the key/value pairs between the mappers and reducers, so the log messages have to be written to a specific log file - check the sample code and the python logging documentation for more details. This Query might also help.
Altri suggerimenti
Hadoop gathers stderr, which can be viewed on hadoop map/reduce status site. So you can just write to stderr.
Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow