Python select () behavior is weird

Question

I'm having trouble understanding select.select behavior. Please consider the following Python program:

def str_to_hex(s): def dig(n): if n > 9: return chr(65-10+n) else: return chr(48+n) r = '' while len(s) > 0: c = s[0] s = s[1:] a = ord(c) / 16 b = ord(c) % 16 r = r + dig(a) + dig(b) return r while True: ans,_,_ = select.select([sys.stdin],[],[]) print ans s = ans[0].read(1) if len(s) == 0: break print str_to_hex(s) 

I saved this in test.py file. If you call it like this:

 echo 'hello' | ./test.py 

then I get the expected behavior: choose never block and all data is printed; the program ends.

But if I run the program interactively, I get the most unwanted behavior. Please consider the following console session:

 $ ./test.py hello [<open file '<stdin>', mode 'r' at 0xb742f020>] 68 

Then the program hangs there; select.select now blocks again. Until I have provided more input or closed the input stream so that the next character (and all the rest) is printed, although there are already characters waiting! Can someone explain this behavior to me? I see something like this in the stream tunneling program I wrote, and it ruins the whole thing.

Thank you for reading!

Answer 1

The read sys.stdin method works at a higher level of abstraction than select . When you execute ans[0].read(1) , python actually reads more bytes from the operating system and internally buffers them. select does not know about this extra buffering; He sees only that everything has been read and is blocked until there is no EOF or more input. You can observe this behavior by running something like strace -e read,select python yourprogram.py .

One solution is to replace ans[0].read(1) with os.read(ans[0].fileno(), 1) . os.read is a lower level interface without any buffering between it and the operating system, so it is better for select .

As an alternative, starting python with the -u command-line option also disables extra buffering.

Answer 2

It is waiting for you to signal EOF (you can do this with Ctrl + D when you use it interactively). You can use sys.stdin.isatty() to check if the script is being executed interactively and process it accordingly using raw_input instead. I also doubt that you need to use select.select at all, why not just use sys.stdin.read ?

 if sys.stdin.isatty(): while True: for s in raw_input(): print str_to_hex(s) else: while True: for s in sys.stdin.read(1): print str_to_hex(s) 

This would make it suitable for both interactive use and stream processing.