The correct way to reuse and close a subprocess object

Question

I have the following code in a loop:

while true: # Define shell_command p1 = Popen(shell_command, shell=shell_type, stdout=PIPE, stderr=PIPE, preexec_fn=os.setsid) result = p1.stdout.read(); # Define condition if condition: break; 

where shell_command is something like ls (it just prints stuff).

I read in various places that I can close / end / close a Popen object Popen various ways, for example.

 p1.stdout.close() p1.stdin.close() p1.terminate p1.kill 

My question is:

• What is the correct way to close a subprocess object when we finish using it?
• Given the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands? Would it be more efficient than opening new subprocess objects every time?

Update

I am still a little confused in the sequence of steps that need to be performed, depending on whether I p1.communicate() or p1.stdout.read() to interact with my process.

From what I understood in the answers and comments:

If I use p1.communicate() , I don’t have to worry about releasing resources, since communicate() will wait for the process to complete, capture the output and close the subprocess object correctly

If I follow the route p1.stdout.read() (which, it seems to me, is suitable for my situation, since the shell command is just for printing), I should call things in the following order:

• p1.wait()
• p1.stdout.read()
• p1.terminate()

Is it correct?

Answer 1

What is the correct way to close a subprocess object when we finish using it?

stdout.close() and stdin.close() do not terminate the process if it does not exit the end of the input or write errors.

.terminate() and .kill() both do the job, with kill will be a bit more "radical" on POSIX systems, since SIGKILL sent, which cannot be ignored by the application. Specific differences are explained, for example, in this blog post . There is no difference in Windows.

Also, don't forget .wait() and close the pipes after killing the process to avoid the zombies and force them to free up resources.

A common case that often occurs are processes that are read from STDIN and write their result to STDOUT, closing when an EOF is encountered. With such programs, it is often wise to use subprocess.communicate :

 >>> p = Popen(["sort"], stdin=PIPE, stdout=PIPE) >>> p.communicate("4\n3\n1") ('1\n3\n4\n', None) >>> p.returncode 0 

This can also be used for programs that print something and exit immediately after:

 >>> p = Popen(["ls", "/home/niklas/test"], stdin=PIPE, stdout=PIPE) >>> p.communicate() ('file1\nfile2\n', None) >>> p.returncode 0 

Given the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands? Would it be more efficient than opening new subprocess objects every time?

I don’t think the subprocess module supports this, and I don’t see what resources can be shared here, so I don’t think it will give you a significant advantage.

Answer 2

The "correct" order:

• Create a stream to read stdout (and a second to read stderr, if you haven't combined them into one).

• Record the commands that will be executed by the child in stdin. If you are not reading stdout at the same time, writing to stdin may be blocked.

• Close stdin (this is a signal for a child element that it can complete by itself when it is done)

• When stdout returns EOF, the child completes. Note that you need to synchronize the stdout read stream and your main stream.

• call wait() to find out if there is a problem and clear the child process

If you need to stop the child process for any reason (perhaps the user wants to exit), you can:

• Close stdin if the child terminates when it reads EOF.

• Kill with terminate() . This is the right solution for child processes that ignore stdin.

• If the child is not responding, try kill()

In all three cases, you must call wait() to clear the dead child process.

Answer 3

Given the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands?

Yes.

 #!/usr/bin/env python from __future__ import print_function import uuid import random from subprocess import Popen, PIPE, STDOUT MARKER = str(uuid.uuid4()) shell_command = 'echo a' p = Popen('sh', stdin=PIPE, stdout=PIPE, stderr=STDOUT, universal_newlines=True) # decode output as utf-8, newline is '\n' while True: # write next command print(shell_command, file=p.stdin) # insert MARKER into stdout to separate output from different shell_command print("echo '%s'" % MARKER, file=p.stdin) # read command output for line in iter(p.stdout.readline, MARKER+'\n'): if line.endswith(MARKER+'\n'): print(line[:-len(MARKER)-1]) break # command output ended without a newline print(line, end='') # exit on condition if random.random() < 0.1: break # cleanup p.stdout.close() if p.stderr: p.stderr.close() p.stdin.close() p.wait() 

Put while True inside try: ... finally: to clean up in case of exceptions. On Python 3.2+, you can use with Popen(...): instead.

Would it be more efficient than opening new subprocess objects every time?

Does it matter in your case? Do not guess. Measure it.

Answer 4

• Depends on what you expect from the process; you should always call p1.wait() to avoid zombies. Other steps depend on the behavior of the subprocess; if it produces any output, you should consume the output (for example, p1.read() ... but it will be a lot of memory), and only then call p1.wait() ; or you can wait a while and call p1.terminate() to kill the process if you think that it is not working properly, and it is possible to call p1.wait() to clear the zombies.

Alternatively, p1.communicate(...) will do the processing if io is waiting for you (rather than killing).

• Subprocess objects should not be reused.