How to register multiple threads in different log files?

Question

I have a JAVA class that runs various threads with unique identifiers. Each thread must enter a unique log file named after ID.log.

Since I only get a unique identifier at runtime, I have to programmatically configure Log4J:

// Get the jobID myJobID = aJobID; // Initialize the logger myLogger = Logger.getLogger(myJobID); FileAppender myFileAppender; try { myFileAppender = new FileAppender(new SimpleLayout(), myJobID + ".log", false); BasicConfigurator.resetConfiguration(); BasicConfigurator.configure(myFileAppender); } catch (IOException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } 

Now it works great if I start tasks sequentially - but when I start 2 threads (of the same class) at the same time, both logs are created, but the logs are mixed: the second thread is registered in both the first and second logs.

How can I make sure each instance is unique? I already tried to specify a unique name for each instance of the log, but did not change anything.

Answer 1

There is a special appender SiftingAppender in the logbook , which provides a very good solution for the type of problems you describe. SiftingAppender can be used to split (or sort) the log according to any runtime attribute, including stream identifier.

Answer 2

For log4j v2, you can use RoutingAppender to dynamically route messages. You can put the value of the 'threadId' key in the ThreadContext map, and then use this identifier as part of the file name. There is an example that I easily applied for the same purpose as yours. See http://logging.apache.org/log4j/2.x/faq.html#separate_log_files

Be careful when placing values ​​in the ThreadContext: "The child thread automatically inherits a copy of its parent’s associated diagnostic context." Therefore, if you put the value of the 'threadId' key in the parent thread and eventually create several threads from it, then all child threads will inherit the value of the 'threadId' value. I was unable to simply override this value using put() again - you need to use ThreadContext.clear() or explicitly remove() value from the thread context map.

Here is my working log4j.xml:

 <?xml version="1.0" encoding="UTF-8"?> <configuration status="WARN"> <properties> <property name="logMsgPattern">%d{HH:mm:ss} %-5level - %msg%n</property> <property name="logDir">test logs</property><!-- ${sys:testLogDir} --> </properties> <appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="${logMsgPattern}"/> </Console> <Routing name="Routing"> <Routes pattern="$${ctx:threadId}"> <Route> <RollingFile name="RollingFile-${ctx:threadId}" fileName="${logDir}/last-${ctx:threadId}.log" filePattern="${logDir}/%d{yyyy-MM-dd}/archived_%d{HH-mm}-${ctx:threadId}.log"> <PatternLayout pattern="${logMsgPattern}"/> <Policies> <OnStartupTriggeringPolicy /> </Policies> </RollingFile> </Route> </Routes> </Routing> </appenders> <loggers> <root level="debug"> <appender-ref ref="Console" level="debug" /> <appender-ref ref="Routing" level="debug"/> </root> </loggers> </configuration> 

Answer 3

An approach

@havexz is pretty good: writes everything to the same log file and uses nested diagnostic contexts .

If you are worried about multiple JVMs that write the same FileAppender, I would suggest two things:

• using SLF4J as a facade for logging
• using logback as a logging implementation, in a reasonable way

In reasonable mode, FileAppender will safely write to the specified file, even in the presence of other instances of FileAppender running in different JVMs, potentially running on different hosts.

Answer 4

Here is a snippet of routing code from the working log4j.xml file.

 <Appenders> <Console name="ConsoleAppender" target="SYSTEM_OUT"> <PatternLayout> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %-50logger{4}: %msg%n</pattern> </PatternLayout> </Console> <Routing name="RoutingAppender"> <Routes pattern="${ctx:logFileName}"> <!-- This route is chosen if ThreadContext has a value for logFileName. The value dynamically determines the name of the log file. --> <Route> <RollingFile name="Rolling-${ctx:logFileName}" fileName="${sys:log.path}/${ctx:logFileName}.javalog" filePattern="./logs/${date:yyyy-MM}/${ctx:logFileName}_%d{yyyy-MM-dd}-%i.log.gz"> <PatternLayout> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %-50logger{4}: %msg%n</pattern> </PatternLayout> <Policies> <TimeBasedTriggeringPolicy interval="6" modulate="true" /> <SizeBasedTriggeringPolicy size="10 MB" /> </Policies> </RollingFile> </Route> <!-- This route is chosen if ThreadContext has no value for key logFileName. --> <Route key="${ctx:logFileName}" ref="ConsoleAppender" /> </Routes> </Routing> </Appenders> <loggers> <root level="debug"> <appender-ref ref="RoutingAppender" level="debug" /> </root> </loggers> 

The key "logFileName" can be added to the stream context map in the run () method of the Runnable class as follows:

 public class SomeClass implements Runnable{ private int threadID; public SomeClass(int threadID){ this.threadID=threadID; } @Override public void run() { String logFileName = "thread_log_"+ threadID; ThreadContext.put("logFileName", logFileName); //Some code ThreadContext.remove("threadId"); } } 

In addition, you need to import the correct log4j packages, as shown below.

 import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.ThreadContext; 

Please note that the next import will not work. LogManager and Logger should also come from org.apache.logging.log4j.

 import org.apache.log4j.LogManager; import org.apache.log4j.Logger; import org.apache.logging.log4j.ThreadContext; 

Answer 5

How to add a static instance counter variable to your class. Then you need a synchronized method that increments the counter for each object created and creates the log file name from this value. Something like that:

 class yourClass { private static int cnt = 0; public yourClass(){ ... initLogger(); } private synchronized initLogger(){ yourClass.cnt++; myJobid = yourClass.cnt; //include your logging code here } } 

Answer 6

As far as I can tell, the ThreadLocal API was designed to do what you describe.

In a code like the one below, logs will be installed on each thread, each of which uses its own (for threads) FileAppender:

 /** * usage: threadLocalLogger.get().info("hello thread local logger") */ static ThreadLocal<Logger> threadLocalLogger = newThreadLocalLogger("myJobId"); private static ThreadLocal<Logger> newThreadLocalLogger(final String myJobID) { return new ThreadLocal<Logger>() { @Override protected Logger initialValue() { return logger(myJobID, Thread.currentThread().getId()); } }; } private static Logger logger(String myJobID, long threadId) { // Initialize the logger String loggerId = myJobID + "-" + threadId; Logger myLogger = Logger.getLogger(loggerId); FileAppender myFileAppender; try { myFileAppender = new FileAppender(new SimpleLayout(), loggerId + ".log", false); BasicConfigurator.resetConfiguration(); BasicConfigurator.configure(myFileAppender); } catch (IOException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } return myLogger; }