Where does Hadoop store the logs of YARN applications?

Question

I run the basic example of Hortonworks' yarn application example. The application fails and I want to read the logs to figure out why. But I can't find any files at the expected location (/HADOOP_INSTALL_FOLDER/logs) where the logs of my mapreduce jobs are stored.

Does anybody know where yarn stores the non-mapreduce log files? Do I have to configure a special directory in the xml files?

Answer 1

The container logs should be under yarn.nodemanager.log-dirs:

Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

Answer 2

Log-aggregation has been implemented in YARN, because of which the log file locations will vary when compared with Hadoop 1.

You can get the logs of your application in two ways, WebUi and Command line access.

Please go through the below document which gives you a very clear information on this log-aggregation implementation on YARN.

http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

Thanks to the blog author, Vinod Kumar Vavilapalli.