Direct piping with python

Question

I have two files, the first of which is fizz

#!/usr/bin/python import time print 'started' time.sleep(3) print 'ended' 

next of which: bar

 #!/usr/bin/python import sys for line in sys.stdin: print line 

When I run the team ./fizz | ./bar ./fizz | ./bar , I expect it to print started , and then wait 3 seconds and type ended , but what really happens is that it prints started and ended at the same time after 3 seconds. Is there a way to achieve the desired behavior? Thanks

Answer 1

Good question. This is a little harder than necessary.

The problem is really in bar , in particular sys.stdin buffered. I tried opening sys.stdin with a smaller buffer size and using python -u , but that did not work. The manpage has the following:

  -u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode. Note that there is internal buffering in xread‐ lines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop. 

In the end, this is what worked for me:

 #!/usr/bin/python import sys import os while True: line = sys.stdin.readline() if not line: break sys.stdout.write(line) # or print, doesn't matter. 

Answer 2

Now that it’s clear that the problem is on the receiving side, I present an alternative that I like to use:

 #!/usr/bin/python import sys import os for line in iter(sys.stdin.readline, ''): sys.stdout.write(line) # \n included in line 

iter(func, sentinel) calls func() for each iteration and ends if the result of the function == sentinel .

Answer 3

There are two problems:

• print "something" in ./foo does not ./foo its stdout buffer if it is redirected (in this case to the pipe), i.e. when stdout is not connected tty-like device , for example, to an interactive console
• for line in sys.stdin: may try to read several lines at a time

You can fix it as follows:

 $ PYTHONUNBUFFERED=1 ./foo | ./bar 

Where ./bar :

 #!/usr/bin/python import sys for line in iter(sys.stdin.readline, ''): print line, 

those. make ./foo stdout unbuffered ( -u option ) and read the input in ./bar line by line as suggested by @Eduardo Ivanec answer .

Alternatively, you can call sys.stdout.flush() in ./foo instead of making it stdout unbuffered as suggested by @kev's answer .