Suppress multiple messages with the same content in Python logging module AKA log compression

Question

By design, my application sometimes creates repeated errors that populate the log file and make it annoying to read. It looks like this:

WARNING:__main__:CRON10: clock unset or no wind update received in 60 sec -> supressed rrd update WARNING:__main__:CRON10: clock unset or no wind update received in 60 sec -> supressed rrd update WARNING:__main__:CRON10: clock unset or no wind update received in 60 sec -> supressed rrd update WARNING:__main__:CRON10: clock unset or no wind update received in 60 sec -> supressed rrd update 

How can I use the Python logging module to suppress duplicate messages and output something more rsyslog style ( http://www.rsyslog.com/doc/rsconf1_repeatedmsgreduction.html ):

 WARNING:__main__:CRON10: clock unset or no wind update received in 60 sec -> supressed rrd update --- The last message repeated 3 times 

Is there a way to expand logging or do I need to write my own log?

The code I use for logging is:

 logging.basicConfig(format='%(asctime)s %(message)s') logging.basicConfig(level=logging.info) logger = logging.getLogger(__name__) hdlr = logging.FileHandler(LOGFILE) hdlr.setFormatter(formatter) logger.addHandler(hdlr) 

Any ideas on this?

Answer 1

You can create a logging.Filter that will track the last recorded record and filter out any duplicate (similar) records, something like:

 import logging class DuplicateFilter(logging.Filter): def filter(self, record): # add other fields if you need more granular comparison, depends on your app current_log = (record.module, record.levelno, record.msg) if current_log != getattr(self, "last_log", None): self.last_log = current_log return True return False 

Then just add it to your logger / handler (i.e. hdlr.addFilter(DuplicateFilter()) ) or the root log to filter all logs by default. Here is a simple test:

 import logging logging.warn("my test") logging.warn("my repeated test") logging.warn("my repeated test") logging.warn("my repeated test") logging.warn("my other test") logger = logging.getLogger() # get the root logger logger.addFilter(DuplicateFilter()) # add the filter to it logging.warn("my test") logging.warn("my repeated test") logging.warn("my repeated test") logging.warn("my repeated test") logging.warn("my other test") 

This will print:

  WARNING: root: my test
 WARNING: root: my repeated test
 WARNING: root: my repeated test
 WARNING: root: my repeated test
 WARNING: root: my other test
 WARNING: root: my test
 WARNING: root: my repeated test
 WARNING: root: my other test