Using tee to get real-time print statements from python

Question

I have a python script that looks something like this:

for item in collection: print "what up" #do complicated stuff that takes a long time. 

In bash, I ran this script by doing the following:

 $ python my.py | tee my_file.txt 

However, everything I see in bash is an empty string until the program terminates. Then all print statements appear on one.

Is this the expected work of the tee? Can I use a tee to view output in real time?

Answer 1

Python, like many other programs, tries to minimize the number of times it calls the write system call. He does this by collecting the output of several print statements before he actually writes them to his standard output file. This process is called output buffering.

When Python is connected to a terminal, it does not buffer its output. This makes sense because the person at the terminal wants to immediately see the way out.

When Python writes a file (or pipe), it buffers its output. This also makes sense since no one will see the output until the process is complete.

You can defeat this optimization by calling sys.stdout.flush() whenever you want to force Python to write its buffer output to its standard output file.

In your case, try the following:

 import sys ... for item in collection: print "what up" sys.stdout.flush() #do complicated stuff that takes a long time.