Python: reading a file continuously, even after it hangs

Question

I have a simple python script where I read logfile all the time (same as tail -f )

 while True: line = f.readline() if line: print line, else: time.sleep(0.1) 

How can I make sure that I can still read the log file after it is rotated using logrotate?

i.e. I need to do the same thing as tail -f .

I am using python 2.7

Answer 1

While you plan to do this only on Unix, the most reliable way is probably to check that the open file still refers to the same i-node as the name and reopen it when it is no longer. You can get the i-number of the file from os.stat and os.fstat in the st_ino field.

It might look like this:

 import os, sys, time name = "logfile" current = open(name, "r") curino = os.fstat(current.fileno()).st_ino while True: while True: buf = current.read(1024) if buf == "": break sys.stdout.write(buf) try: if os.stat(name).st_ino != curino: new = open(name, "r") current.close() current = new curino = os.fstat(current.fileno()).st_ino continue except IOError: pass time.sleep(1) 

I doubt this works on Windows, but since you are speaking in terms of tail , I assume this is not a problem. :)

Answer 2

You can do this by tracking where you are in the file and reopening it when you want to read. When the log file rotates, you will notice that the file is smaller, and from the moment you reopen it, you also handle any disconnection.

 import time cur = 0 while True: try: with open('myfile') as f: f.seek(0,2) if f.tell() < cur: f.seek(0,0) else: f.seek(cur,0) for line in f: print line.strip() cur = f.tell() except IOError, e: pass time.sleep(1) 

This example hides errors, such as a file not found, because I am not sure about the details of logrotate, such as short periods of time when the file is not available.

Answer 3

Thanks to the answers of @tdelaney and @ Dolda2000, I ended up with what follows. It should work with both Linux and Windows, as well as handle the parameters logrotate copytruncate or create (respectively copy, crop the size to 0 and move and then recreate the file).

 file_name = 'my_log_file' seek_end = True while True: # handle moved/truncated files by allowing to reopen with open(file_name) as f: if seek_end: # reopened files must not seek end f.seek(0, 2) while True: # line reading loop line = f.readline() if not line: try: if f.tell() > os.path.getsize(file_name): # rotation occurred (copytruncate/create) f.close() seek_end = False break except FileNotFoundError: # rotation occurred but new file still not created pass # wait 1 second and retry time.sleep(1) do_stuff_with(line) 

The limitation when using the copytruncate parameter is that if lines are attached to the file during sleep mode, and rotation occurs before , the last lines will be "lost" (they will still be in the "old" log file, but I don’t see a decent the way to "follow" this file to finish reading it). This restriction is not related to the "move and create" create option, because the descriptor f will still point to the renamed file, and therefore the last lines will be read before the descriptor is closed and reopened.