Custom log4j appender in Hadoop 2

Question

How to specify custom log4j appender in Hadoop 2 (amazon emr)?

Hadoop 2 ignores the log4j.properties file, which contains the custom appender, overriding it with the internal log4j.properties file. There is a flag -Dhadoop.root.loggerthat indicates the registration threshold , but this does not help for the user application.

Answer 1

1. To change log4j.properties to a node name, you can change / home / hadoop / log 4j.properties.

2. log4j.properties , , .

2.1 ssh ( EMR u , ssh ). ssh to hadoop slave

2.2 -log4j.properties :

jar uf/home/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar container -log4j.properties

Answer 2

, , , . log4j.properties, Hadoop ( , YARN). Cloudera, .

: /etc/hadoop/conf/log4j.properties ( )

log4j.properties, java-. , Java-, YARN/Map Reduce. , , , , (, ) , . , .

, hadoop, - .

, , ( ). , .

: etc/hadoop/conf/container-log4j.properties ( node)

, , , hadoop.

Hadoop/YARN - , , , . - container-log4j.properties, , :

log4j.logger.org.apache.hadoop.mapreduce=WARN
log4j.logger.org.apache.hadoop.mapred=WARN
log4j.logger.org.apache.hadoop.yarn=WARN
log4j.logger.org.apache.hadoop.hive=WARN
log4j.security.logger=WARN

, , hasoop-yar-server-nodemanager-stuff.jar( uriah kremer). , log4j-properties, /etc/hadoop/conf/container-log4j.properties, YARN. !

. , , -log4j , YARN nodemanager . , , , -Dlog4j.configuration=PATH_TO_FILE , .

Answer 3

Look for hasoop-config.sh in the deployment. This is the script received before the hadoop command was executed. I see the following code in hasoop-config.sh, see if this change helps.

HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=${HADOOP_ROOT_LOGGER:-INFO,console}"