How to limit the number of identical log messages?

Question

I use the logging module to warn about problems with some procedures. These procedures can be started several times before returning to normal operation (for example, repeated requests to the API that are not executed, but eventually pass). Each failed call causes a log entry.

Is there a way to limit the number of identical log messages?
I would like this limit to be triggered after nidentical messages that were displayed, and then maybe tell me what nmore was created (so as not to clutter the log file) and reset after receiving the recovery log. This is an ideal scenario - I am looking at how to approach a problem in order to start with it.

The closest I found is a conditional version of the magazines , but I don’t see how this could be adapted to my case.
 Another possibility is to set a limit at the syslog level (in rsyslogor syslog-ng), but this is a "per process" option, so I can lose useful logs (those that fall between loops)

Answer 1

Use logging.Filter!

DuplicateFilter accepts two regex patterns or a string (which should be compiled into one) that should match the string you want to filter and you want to reset the filter to.

import logging
import os
import re
import sys

from sre_parse import Pattern


class DuplicateFilter(logging.Filter):
    def __init__(self, match_against, reset_at_message, hide_at_count=5, name=''):
        super(DuplicateFilter, self).__init__(name)

        if isinstance(match_against, Pattern):
            self.match_against = match_against
        else:
            self.match_against = re.compile(match_against)

        if isinstance(reset_at_message, Pattern):
            self.reset_at_message = reset_at_message
        else:
            self.reset_at_message = re.compile(reset_at_message)

        self.hide_at_count = hide_at_count

        self.count = 0

    def filter(self, record: logging.LogRecord):
        _ = super(DuplicateFilter, self).filter(record)
        if not _:
            return _

        msg = record.getMessage()

        if self.match_against.match(msg):
            self.count += 1

            if self.count >= self.hide_at_count:
                return False

        elif self.reset_at_message.match(msg):
            record.msg = os.linesep.join([
                '{:d} more generated'.format(self.count - self.hide_at_count),
                record.msg
            ])
            self.count = 0

        return True

handler = logging.StreamHandler(sys.stdout)
handler.addFilter(DuplicateFilter('Filter me!', 'Reset at me'))

logging.basicConfig(level='INFO', handlers=[handler, ])

log = logging.getLogger()

for _ in range(10):
    log.info('Filter me!')

log.info('Reset at me')

for _ in range(3):
    log.info('Filter me!')

This is the resulting log:

INFO:root:Filter me!
INFO:root:Filter me!
INFO:root:Filter me!
INFO:root:Filter me!
INFO:root:5 more generated
Reset at me
INFO:root:Filter me!
INFO:root:Filter me!
INFO:root:Filter me!

Only the "5 more generated" pending messages are probably not what you want, but hopefully this is a good starting point.