hadoop flume log4j configuration

Question

If you run a hadoop flume node, as default it generates logs under /var/log/flume using log4j. The files will look like

/var/log/flume/flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log

According to the flume user guide here, the only way to change the flume log configuration is via flume-daemon.sh which runs flume node using the Flume Environment Variables like:

export FLUME_LOGFILE=flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log
export FLUME_ROOT_LOGGER="INFO,DRFA"
export ZOOKEEPER_ROOT_LOGGER="INFO,zookeeper"
export WATCHDOG_ROOT_LOGGER="INFO,watchdog"

The questions are:

1. if I want to change the log level from INFO to DEBUG, this is the only place to do it?
2. Is there a configuration somewhere I can do this?
3. what about I want to set some packages' log level to DEBUG while others stay INFO?

Answer 1

Noticed that under /etc/flume/conf.empty, there is a log4j.properties. Copied it to /etc/flume/conf, restart the flume node service, the log4j.properties file starts taking effect.

The overriding order is like flume-env.sh->flume-daemon.sh->log4j.properties.

eg. if you set up your flume_root_logger to DEBUG in flue-daemon.sh, it will override whatever setting you have for root_logger in your log4j.properties.

Answer 2

Check if the log4j.properties or log* related files exist to set the variables -- which will also allow you to check and have some components of the logging part do excessive/DEBUG while other do INFO.