Hadoop Logging Level Configuration for WARN

Question

I tried many ways to set the logging level in the Hadoop WARN, but failed every time. First, I tried to configure the log4j.properties file by simply replacing “INFO” with “WARN”. No result.

Then I tried to give the Hadoop UNIX commands (according to http://hadoop.apache.org/common/docs/current/commands_manual.html#daemonlog ):

$ hadoop daemonlog -setlevel

Is it possible that you really need to change the SOURCE CODE to make it work? Logging is often quite simple to control, in most cases a small adjustment to the logging properties usually does this ...

Answer 1

I prefer to use

HADOOP_ROOT_LOGGER = WARN, DRFA

at hadoop-env.sh

or you can use hasoop.root.logger in log4j.properties

DRFA allows logs to go to File Appender, and not to Console -> System.err / out.

Answer 2

The default log level can be adjusted by changing the hadoop.root.logger property in the conf/log4j.properties configuration file. Note that you will need to do this for each node in your cluster.

Example line in conf/log4j.properties :

 hadoop.root.logger=WARN,console 

Answer 3

The Apache hadoop documentation is a bit misleading. If you are debugging problems, you can change the log level on the fly using the steps below. You must specify the package name, not the file name.

Example: for Namenode: hadoop daemonlog -setlevel lxv-centos-01: 50070 org.apache.hadoop.hdfs.server.namenode DEBUG

For Resourcemanager yne daemonlog -setlevel lxv-centos-01: 8088 org.apache.hadoop.yarn.server.resourcemanager DEBUG

When processes are restarted, the above setting disappears. This is a temporary fix for debugging problems.

Answer 4

To dynamically change log levels so that a daemon restart is not required, use the daoonlog utility for hadoop.

  hadoop daemonlog -setlevel hostname:port className logLevel 

For example, to change the log level of data logs in WARN.

  hadoop daemonlog -setlevel hostname:50075 org.apache.hadoop.hdfs.server.datanode.DataNode WARN