Bash pipe to python

Question

I need to absorb the output of a bash command through a channel in real time. For instance,

for i in $(seq 1 4); do echo $i; sleep 1; done | ./script.py 

Where script.py has this

 for line in sys.stdin.readlines(): print line 

I expect the sequence to be printed when it becomes available, but the python script waits for the bash script to complete before continuing.

I looked at this answer, but that did not solve my problem. How can i achieve this in python?

Answer 1

The first problem is that readlines reads all the lines in the list. He cannot do this until all lines are present, which will not be until stdin reaches EOF.

But actually you do not need a list of lines, but just a few lines. And a file like sys.stdin is already so iterable. And it's lazy that generates one row at a time as soon as they are available, instead of waiting to generate them all at once.

So:

 for line in sys.stdin: print line 

Whenever you find yourself readlines , ask yourself if you really need it. The answer will always be negative. (Well, except when you want to call it with an argument or on some defective object other than a file.) See Readlines, which is considered silly for more.

---

But meanwhile, there is a second problem. It’s not that Python is buffering its stdin or that another process is buffering its stdout , but the file object iterator itself is doing internal buffering, which may (depending on your platform, but on most POSIX platforms, it usually will prevent you from getting there until the first line before EOF, or at least until many lines are read.

This is a known issue with Python 2.x, which was fixed in 3.x, ^* but it will not help you if you do not want to upgrade.

The solution is mentioned in the command line and environment documents and in the man page of most systems, but the documentation is buried in the middle of -u flag :

Note that inside xreadlines (), readlines (), and file and object iterators, there is internal buffering ("for a line in sys.stdin") that is not affected by this option. To get around this, you will want to use "sys.stdin.readline ()" inside the "1:" loop.

In other words:

 while True: line = sys.stdin.readline() if not line: break print line 

Or:

 for line in iter(sys.stdin.readline, ''): print line 

---

For another problem, in this answer , Alex Martelli points out that you can always just ignore the sys.stdin and rec fdopen file descriptor. This means that you get a wrapper around POSIX fd instead of the C stdio handle. But this is neither necessary nor sufficient for this question, because the problem is not with C stdio buffering, but with how file.__iter__ buffering interacts with it.

---

_{* Python 3.x no longer uses the Stdio library buffering; it does everything on its own in io types, which means that the iterator can simply use the same buffer, the file itself using the file. Although io is also available on 2.x, this is not the standard thing you get for open or for file descriptors stdio, so this does not help here.}

Answer 2

With Python 2.7.9 (and probably all Python before 3.x), this does what you expect:

 #!/usr/bin/python import sys while True: line=sys.stdin.readline() if not line: break print line 

You can also do:

 #!/usr/bin/python import sys for line in iter(sys.stdin.readline, ''): print line 

In Python 3.4.3, you can do what abarnert offers:

 #!/usr/local/bin/python3 import sys for line in sys.stdin: print(line) 

You can also re-open sys.stdin with the io class as it uses Python 3:

 #!/usr/bin/python import sys, io for line in io.open(sys.stdin.fileno()): print(line) 

The first, second, and last methods work on Python 2.7.6 and 2.7.9 and Python 3.4.3 on OS X; third method, only in Python 3.

Answer 3

The current most-answered answer does not actually answer the question, since it does not output output as it flows. Something like the code below should do what you want:

 import sys def readline(): while True: res = sys.stdin.readline() if not res: break yield res for line in readline(): print line 

Here, instead of waiting for the lines to be read to build the list, we read one line and then get the value. And we just continue to consume input and yield until the end of the stream is signaled by an empty return from sys.stdin.readline ().