How to suppress duplicate messages in log4j2

Question

I'm new to Java and log4j2, so sorry, maybe a weird question. My problem is as follows. I wrote an application that uses log4j2 for logging. The program analyzes the data and writes a warning in case this line cannot be analyzed as desired. Sometimes a program receives a lot of unexpected lines and, thus, logs the same error message all the time. So the question is how to avoid re-entering the same error message. Instead, for example, to see the same error message 2000 times in the log file, I would like to have a hint in the log file that this error message was written x-times. My current log4j2 configuration file is as follows:

<?xml version="1.0" encoding="UTF-8"?> <Configuration> <Properties> <Property name="pattern">%d{DEFAULT} %-5p %-18.18c %4.4L [%-15.15t] %m%n</Property> </Properties> <Appenders> <Console name="STDERR" target="SYSTEM_ERR"> <PatternLayout pattern="${pattern}" /> </Console> </Appenders> <Loggers> <Root level="warn"> <AppenderRef ref="STDERR" /> </Root> </Loggers> </Configuration> 

Answer 1

Unfortunately, I don’t have the code for me right now, but here is how you can solve it (and maybe there will already be a solution).

You can create a new appender that will register asynchronously on your file / console.

In the application, you can implement aggregation yourself. Whenever a login comes in, instead of writing it out immediately, check if you've seen the message before. If you did, instead of logging it, increase your counter.

In the end, convert these aggregated log events to an aggregated message and process it, not your thousands of log events.

This can be resolved (for example) with a second worker on your asynchronous application database. Usually (I look at the lock, but I suppose it will be quite similar to you), asynchronous applications will end up adding events to the queue that will be processed by another thread that runs from this queue.

You may have another queue for your aggregated messages, and the worker who works will say every 5 seconds. If in the same 5 seconds the same message has been viewed several times, your worker can aggregate this message and put it in a queue for processing so that it eventually ends up in the logs.

I hope this helps,

Arthur

PS: As for your configuration, just replace the console appender with your custom aggregation add-on. If you want a console, just extend this appender and do as you want.

Answer 2

Of course, you can create a log4j2 plugin that does this, but I would say that the logging library is not the best place to solve this problem.

I think you will get a much simpler code if you follow the previous message that you registered in your application. If the next message is the same, then do not register, but increase the counter. When you press another message, you register the counter value.

(This can be made more general by using Dequeue to track multiple posts.)