Python: logging module for print statements: duplicate log entries

Question

UPDATE: Scroll down to the EDIT section (4) for an almost fully working version. Also delete EDIT SECTION (1) because this post is too long otherwise, and this is probably the least useful section. The link that was originally in the EDIT SECTION (1), below.

How to duplicate sys.stdout file to log file in python?

A long time seeker here, but the first time asks a question.

EXPLANATION:

I need the fingerprints to be redirected to the log because I enable C code through system calls that use print statements to log messages. There are also piles of old Python code written by my peers that is invoked, which also uses print statements for debugging.

Ultimately, I want this to be able to handle logging.info ('message') for my updated code, but also be able to redirect print statements through the built-in logging module for code I, which I either cannot change, or just another did not get to the update.

Below is an example of the code I came up with to briefly demonstrate my problems.

QUESTIONS:

• I use the settings below for my logs, but every time I print, I get duplicate entries in my log (and an empty line). Can anyone explain why this is happening?
• It would be nice to find out the best setting for logging, so the format operator includes the correct module name when I redirect the print command through logging.
• My use of this Tee (object) class seems to interrupt the situation sometimes. See Supporting Information Section below.

MY CODE:

Edit: Initially setuplog.Tee. init contains if os.path.exists(LOGNAME): os.remove(LOGNAME) . It has been removed and placed in base.py.

setuplog.py:

 #!/usr/bin/python import sys import os import logging import logging.config LOGNAME = 'log.txt' CONFIG = { 'version': 1, 'disable_existing_loggers': True, 'formatters': { 'simple': { 'format': '%(module)s:%(thread)d: %(message)s' }, }, 'handlers': { 'console': { 'level': 'NOTSET', 'class': 'logging.StreamHandler', 'formatter': 'simple' }, 'file': { 'level': 'NOTSET', 'class': 'logging.FileHandler', 'formatter': 'simple', 'filename': LOGNAME }, }, 'root': { 'level': 'NOTSET', 'handlers': ['console', 'file'] }, } class Tee(object): def __init__(self): logging.config.dictConfig(CONFIG) self.logger = logging.getLogger(__name__) self.stdout = sys.stdout sys.stdout = self def __del__(self): sys.stdout = self.stdout def write(self, data): self.logger.info(data) 

base.py:

 #!/usr/bin/python import sys import os import logging # My modules. import setuplog #import aux2 LOGNAME = 'log.txt' if os.path.exists(LOGNAME): os.remove(LOGNAME) not_sure_what_to_call_this = setuplog.Tee() print '1 base' logging.info('2 base') print '3 base' os.system('./aux1.py') logging.info('4 base') #aux2.aux2Function() #logging.info('5 base') 

aux1.py:

 #!/usr/bin/python import sys import os import logging import setuplog not_sure_what_to_call_this = setuplog.Tee() def main(): print '1 aux1' logging.info('2 aux1') print '3 aux1' logging.info('4 aux1') if __name__ == '__main__': main() 

aux2.py:

 #!/usr/bin/python import sys import os import logging import setuplog not_sure_what_to_call_this = setuplog.Tee() def aux2Function(): print '1 aux2' logging.info('2 aux2') print '3 aux2' 

Then I run "./base.py" from the shell and create the following output (in the console and in log.txt):

 setuplog:139833296844608: 1 aux1 setuplog:139833296844608: aux1:139833296844608: 2 aux1 setuplog:139833296844608: 3 aux1 setuplog:139833296844608: aux1:139833296844608: 4 aux1 

As you can see, records created with printing are repeated (QUESTION 1). In addition, I need to find a better deal for displaying the module name (QUESTION 2).

SUPPORT FOR INFORMATION FOR QUESTION 3:

From base.py, if I uncomment "import aux2", "aux2.aux2Function ()" and "logging.info (" 5 base "), here is the new output (right from my console, as this is the only place the errors go Python):

 base:140425995155264: 2 base setuplog:140360687101760: 1 aux1 setuplog:140360687101760: aux1:140360687101760: 2 aux1 setuplog:140360687101760: 3 aux1 setuplog:140360687101760: aux1:140360687101760: 4 aux1 base:140425995155264: 4 base aux2:140425995155264: 2 aux2 base:140425995155264: 5 base Exception AttributeError: "'NoneType' object has no attribute 'stdout'" in <bound method Tee.__del__ of <setuplog.Tee object at 0x7fb772f58f10>> ignored 

EDIT SECTION (2):

I play, and such work. Here (again) updated sample code versions.

The reason this "view" works is:

• I think exceptions should be avoided at all costs, and this uses one.
• Log output is now somewhat manual. Let me explain. The value of% (name) s looks as expected, but I have to set this manually. Instead, I would prefer some kind of descriptor that automatically selects a file name or something similar (choosing functions as a bonus?). % (module) s always displays "setuplog" (correctly) for print statements, even if I want the module to report where the print statement came from, and not the module in which my class for direct print requests to the log module.

setuplog.py:

 #!/usr/bin/python import sys import os import logging import logging.config def startLog(name): logname = 'log.txt' config = { 'version': 1, 'disable_existing_loggers': True, 'formatters': { 'simple': { 'format': '%(name)s:%(module)s:%(thread)s: %(message)s' }, }, 'handlers': { 'console': { 'level': 'NOTSET', 'class': 'logging.StreamHandler', 'formatter': 'simple' }, 'file': { 'level': 'NOTSET', 'class': 'logging.FileHandler', 'formatter': 'simple', 'filename': logname }, }, 'root': { 'level': 'NOTSET', 'handlers': ['console', 'file'], }, } logging.config.dictConfig(config) return logging.getLogger(name) class Tee(): def __init__(self, logger): self.stdout = sys.stdout self.data = '' self.logger = logger sys.stdout = self def __del__(self): try: sys.stdout = self.stdout except AttributeError: pass def write(self, data): self.data += data self.data = str(self.data) if '\x0a' in self.data or '\x0d' in self.data: self.data = self.data.rstrip('\x0a\x0d') self.logger.info(self.data) self.data = '' 

base.py:

 #!/usr/bin/python import sys import os import logging # My modules. import setuplog import aux2 LOGNAME = 'log.txt' if os.path.exists(LOGNAME): os.remove(LOGNAME) logger = setuplog.startLog('base') setuplog.Tee(logger) print '1 base' logger.info('2 base') print '3 base' os.system('./aux1.py') logger.info('4 base') aux2.aux2Function() logger.info('5 base') 

aux1.py:

 #!/usr/bin/python import sys import os import logging import setuplog def main(): logger = setuplog.startLog('aux1') setuplog.Tee(logger) print '1 aux1' logger.info('2 aux1') print '3 aux1' logger.info('4 aux1') if __name__ == '__main__': main() 

aux2.py:

 #!/usr/bin/python import sys import os import logging import setuplog def aux2Function(): logger = setuplog.startLog('aux2') setuplog.Tee(logger) print '1 aux2' logger.info('2 aux2') print '3 aux2' 

And the conclusion:

 base:setuplog:139712687740736: 1 base base:base:139712687740736: 2 base base:setuplog:139712687740736: 3 base aux1:setuplog:140408798721856: 1 aux1 aux1:aux1:140408798721856: 2 aux1 aux1:setuplog:140408798721856: 3 aux1 aux1:aux1:140408798721856: 4 aux1 base:base:139712687740736: 4 base aux2:setuplog:139712687740736: 1 aux2 aux2:aux2:139712687740736: 2 aux2 aux2:setuplog:139712687740736: 3 aux2 

SECTION EDIT (3):

Thanks to the wonderful answer to reddit ( http://www.reddit.com/r/learnpython/comments/1kaduo/python_logging_module_for_print_statements/cbn2lef ), I was able to develop work for AttributeError. Instead of using an exception, I converted the class to a singleton.

Here is the code for the updated Tee class:

 class Tee(object): _instance = None def __init__(self, logger): self.stdout = sys.stdout self.data = '' self.logger = logger sys.stdout = self def __new__(cls, *args, **kwargs): if not cls._instance: cls._instance = super(Tee, cls).__new__(cls, *args, **kwargs) return cls._instance def __del__(self): sys.stdout = self.stdout def write(self, data): self.data += data self.data = str(self.data) if '\x0a' in self.data or '\x0d' in self.data: self.data = self.data.rstrip('\x0a\x0d') self.logger.info(self.data) self.data = '' 

EDIT SECTION (4):

It almost works! It works well enough for me to implement it. The only problem now is that the output format is more useful. For example,% (filename) s is setuplog.py for all redirected print statements. It would be much more profitable if% (filename) s was the file from which the print request originated. Any ideas?

In addition, I had to abandon the vocabulary method. The only way I could do this is to configure the logger using Python code.

Last point, take a look at aux3.py. If using os.system rather than a subprocess, the logging order is confused. Does anyone know how to work to still use os.system and get the correct order (so I don't need to change every last os.system to a .Popen subprocess)?

setuplog.py (you can ignore the startDictLog and startFileLog functions because they do not work. However, startCodeLog does!):

 #!/usr/bin/python import sys import os import logging import logging.config def startLog(name, propagate): '''The only point of this function was to enable me to quickly and easily switch how I wanted to configure the logging. So far, I have only been able to get the last way working (startCodeLog).''' #return startDictLog(name) #return startFileLog(name) return startCodeLog(name, propagate) def startDictLog(name): '''Configure logging usinga dictionary.''' LOGNAME = 'loop.log' DEBUGNAME = 'debug.log' config = { 'version': 1, 'disable_existing_loggers': True, 'formatters': { 'bare': { # Added the BARE to distinguish between normal prints # and those that get rerouted. In use, I would format it # such that only the message is printed. 'format': 'BARE: %(message)s' }, 'simple': { 'format': '%(module)s-%(name)s: %(message)s' }, 'time': { 'format': '%(asctime)s-%(filename)s-%(module)s-%(name)s: %(message)s', 'datefmt': '%H:%M:%S' }, }, 'handlers': { 'console': { 'level': 'NOTSET', 'class': 'logging.StreamHandler', 'formatter': 'bare' }, 'normal': { 'level': 'INFO', 'class': 'logging.FileHandler', 'formatter': 'simple', 'filename': LOGNAME }, 'debug': { 'level': 'NOTSET', 'class': 'logging.FileHandler', 'formatter': 'time', 'filename': DEBUGNAME }, }, 'root': { 'level': 'NOTSET', 'handlers': ['console', 'normal', 'debug'], }, } logging.config.dictConfig(config) return logging.getLogger(name) def startFileLog(name): '''Configure logging using a configuration file.''' CONFIGFILE = 'logging.conf' logging.config.fileConfig(CONFIGFILE) return logging.getLogger(name) def startCodeLog(name, propagate): '''Configure logging using this code.''' LOGFILE = 'loop.log' DEBUGFILE = 'debug.log' _logger = logging.getLogger(name) _logger.setLevel(logging.NOTSET) if propagate in [False, 'n', 'no', 0]: _logger.propagate = False _console = logging.StreamHandler() _normal = logging.FileHandler(LOGFILE) _debug = logging.FileHandler(DEBUGFILE) _bare = logging.Formatter('BARE: %(message)s') _simple = logging.Formatter('%(module)s-%(name)s: %(message)s') _time = logging.Formatter('%(asctime)s-%(module)s-%(name)s: %(message)s', datefmt = '%H:%M:%S') # I added _complex only here, to the working way of setting up the configuration, # in hopes that this data may help someone figure out how to get the printout # to be more useful. For example, it not helpful that the filename is # setuplog.py for every print statement. It would be more beneficial to somehow # get the filename of where the print statement originated. _complex = logging.Formatter('%(filename)s-%(funcName)s-%(name)s: %(message)s') # Normally this is set to _bare to imitate the output of the old version of the # scripts I am updating, but for our purposes, _complex is more convenient. _console.setLevel(logging.NOTSET) _console.setFormatter(_complex) # This imitates the format of the logs from versions before I applied this update. _normal.setLevel(logging.INFO) _normal.setFormatter(_bare) # This is a new log file I created to help me debug other aspects of the scipts # I am updating. _debug.setLevel(logging.DEBUG) _debug.setFormatter(_time) _logger.addHandler(_console) _logger.addHandler(_normal) _logger.addHandler(_debug) return _logger class Tee(object): '''Creates a singleton class that tees print statements to the handlers above.''' _instance = None def __init__(self, logger): self.stdout = sys.stdout self.logger = logger sys.stdout = self self._buf = '' # Part of old method in the write function. #self.data = '' def __new__(cls, *args, **kwargs): '''This is the singleton implementation. This avoids errors with multiple instances trying to access the same standard output.''' if not cls._instance: cls._instance = super(Tee, cls).__new__(cls, *args, **kwargs) return cls._instance def write(self, data): # This method doesn't work with how I had to implement subprocess. #self.data = data.rstrip('\r\n') #if self.data: # self.logger.info(self.data) # Also doesn't seem to work with my subprocess implementation. #self.data += data #self.data = str(self.data) #if '\x0a' in self.data or '\x0d' in self.data: # self.data = self.data.rstrip('\x0a\x0d') # self.logger.info(self.data) # self.data = '' # Only way I could get it working with my subprocess implementation. self._buf = ''.join([self._buf, data]) while '\n' in self._buf: line, _, tail = self._buf.partition('\n') if line.strip(): self.logger.info(line) self._buf = tail 

base.py:

 #!/usr/bin/python import sys import os import logging import subprocess # My modules below. import setuplog import aux2 # It is assumed that this script will be executed via the command line, # and each time we run it, we want new log files. LOGNAME = 'loop.log' DEBUGNAME ='debug.log' if os.path.exists(LOGNAME): os.remove(LOGNAME) if os.path.exists(DEBUGNAME): os.remove(DEBUGNAME) # It seems more convenient to store the logging configuration elsewhere, # hence my module setuplog. logger = setuplog.startLog('', True) logger = setuplog.startLog('base', False) # This initializes sys.stdout being redirected through logging. Can anyone # explain how calling this class achieves this? I am a bit fuzzy on my # understanding here. setuplog.Tee(logger) # Test to see how it works in this module. print '1 base' logger.info('2 base') print '3 base' # Below shows how to get logging to work with scripts that can # not be modified. In my case, I have C code that I don't want to modify, # but I still need to log it output. # I will have to go through the old code and change all os.system calls # to instead utilize subprocess. Too bad because this will take some time # and be "busy work", but at least it works. What can ya do, os.system is # depreciated anyway. The positive side is that only the root application # needs to change its system calls to use subprocess. The scipts that it # calls upon can remain untouched. aux1_py_path = '"%s"' % os.path.join(os.path.dirname(__file__), 'aux1.py') #os.system(aux1_py_path) # Example to show how os.system doesn't work. print 'aux1_py_path:', aux1_py_path sys_call = subprocess.Popen(aux1_py_path, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) sys_stdout, sys_stderr = sys_call.communicate() print sys_stdout print sys_stderr # This is to ensure that the order of the logging events is correct. With an # old method I used (not with logging, simply redirecting stdout into an # opened file), the order would often get messed up. logger.info('4 base') # This example is to show how to get logging to work with Python scripts I am # willing to modify, at least partially. See aux2.py. It is simply a print # statement, followed by a logging statement, followed by another print # statement. They all output properly with this method. aux2.aux2Function() # Again to ensure the order of events is correct. logger.info('5 base') 

aux1.py:

 #!/usr/bin/python import logging import subprocess import os def main(): '''We expect the print statements to go through, as they are being sent to logging. However, these logging statements do nothing as no logger has been instantiated. This is the behavior we should expect, as this script mimics a script that we would not modify, so it would not have logging calls anyway.''' print '1 aux1' logging.info('2 aux1') print '3 aux1' logging.info('4 aux1') # Here, neither option works unless the root of all these calls # was made with subprocess and in the script that called the # tee class. If both conditions are met, subprocess works as it # should. However, os.system returns the print out from the call # out of order. aux3_py_path = '"%s"' % os.path.join(os.path.dirname(__file__), 'aux3.py') #os.system(aux3_py_path) sys_call = subprocess.Popen(aux3_py_path, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) sys_stdout, sys_stderr = sys_call.communicate() print sys_stdout print sys_stderr if __name__ == '__main__': main() 

aux2.py:

 #!/usr/bin/python import sys import os import logging import setuplog def aux2Function(): # The following two lines are not necessary for most functionality. # However, if the following two lines are uncommented, then %(name)s # in the format descriptor of logging will correctly be 'aux2' rather # than base. Both lines are required for this to work. logger = setuplog.startLog('aux2', False) setuplog.Tee(logger) print '1 aux2' logging.info('2 aux2') print '3 aux2' 

aux3.py:

 #!/usr/bin/python import logging def main(): '''See __doc__ for aux1.py. Again, we don't expect the logging.info to work, but that okay because theoretically, this is some script we can't modify that simply generates output with print or print like functions.''' print '1 aux3' logging.info('2 aux3') print '3 aux3' if __name__ == '__main__': main() 

logging.conf (doesn't work):

 [loggers] keys=root [handlers] keys=console,normal,debug [formatters] keys=bare,simple,time [logger_root] level=NOTSET handlers=console,normal,debug [handler_console] level=NOTSET class=StreamHandler formatter=bare args=(sys.stdout,) [handler_normal] level=INFO class=FileHandler formatter=simple args=('loop.log',) [handler_debug] level=DEBUG class=FileHandler formatter=time args=('debug.log',) [formatter_bare] format=%(message)s [formatter_simple] format=%(module)s-%(name)s: %(message)s [formatter_time] format=%(asctime)s-%(filename)s-%(module)s-%(name)s: %(message)s datefmt=%H:%M:%S 

Answer 1

Here is at least a partial answer to your first question. You get these empty lines in every print , because the print statements in Python 2.x potentially call stdout.write() twice, once when the data evaluates the expression in the statement, and then again if the optional new line hasn’t been suppressed Final comma.

In addition, if an INFO level message is sent to the registrar set to the level of 'NOTSET' , the message will also be displayed on sys.stderr by default in accordance with the registrar's documentation - this is why you see console output even when Tee . I do not see duplicate log entries.

To prevent spaces, try using this Tee class definition. Note the modification of the write() method (updated to be single to match your "EDIT SECTION (3)"):

 class Tee(object): _instance = None def __init__(self, logger): self.stdout = sys.stdout self.logger = logger sys.stdout = self def __new__(cls, *args, **kwargs): if not cls._instance: cls._instance = super(Tee, cls).__new__(cls, *args, **kwargs) return cls._instance def __del__(self): sys.stdout = self.stdout def write(self, data): data = data.rstrip('\r\n') if data: # anything left? self.logger.info(data) 

With this and all your other updates in place, it seems to work for me (including the uncommented aux2 text). Are there any problems from your point of view? If so, your initially long question got even more so it should be completely cleared, leaving only the most recent code and focus on any remaining problem.