In our EMR clusters, we use custom log4j-appenders and log4j.properties so that we can forward logs to Splunk and let us do the magic that the provided libraries and configurations do not know how to do this.
In EMR 3.x, we did this using the bootstrap action:
- Download from s3 our custom bank banner log4j, log4j.properties, container -log4j.properties, which we have configured.
- Place our custom banner app4j appender in the yarn lib directory in / Home / Hadoop / share / Hadoop / thread / Library /.
- Update the Hadoop class path to use our custom log4j application
- Paste our modified container-log4j.properties file into hadoop-yarn-server-nodemanager.jar in / Home / Hadoop / share / Hadoop / thread /
All of this worked and allowed us to use our application across all Hadoop processes.
We tried to switch to the release of EMR version v4, but we noticed that there were serious changes in the fact that bootstrap actions are performed when there was no installed hadoop yarn (the path / usr / lib / hasoop -narn / does not work), therefore there is no hasoop -yarn-server-nodemanager.jar to change since it is not yet installed (we modify the jar using the command: jar uf / usr / lib / hadoop-yarn / hadoop-yarn-server-nodemanager.jar container-log4j.properties) and not the lib folder into which we can put our custom log4j-appender.
How can we make these changes to EMR 4.x to allow our own registration?