Python: non-blocking + non-existent process

Question

I would like to create a parent process that will create many child processes. Since the parent process is responsible for creating the child process, the parent process does not care about the status of the children.

Since subprocess.call is blocking, it does not work. Therefore, I use subprocess.Popen to replace the call. However, Popen will generate a zombie (non-existent) process as soon as the child finishes ( Link ).

Is there any way to solve this problem?

Thank you in advance

Answer 1

There are many ways to handle this. The key point is that zombie / non-existent processes exist, so that the parent process can collect its statuses.

• As the creator of the process, you can declare your intention to ignore the status. The POSIX method must set the SA_NOCLDWAIT flag (using sigaction ). This is a bit of a pain in Python; but most Unix-like systems let you simply ignore SIGCHLD / SIGCLD (spelling varies from one Unix-like system to another), which is easy to do in Python:

  import signal

  signal.signal(signal.SIGCHLD, signal.SIG_IGN)

• Or, if this is not available for any reason or does not work on your system, you can use the old stunt: do not break out only once, plug twice. In the first child, the plug of the second child; in the second child, use execve (or similar) to run the desired program; and then in the first child exit (with _exit ). In the original parent, use wait or waidpid or whatever the OS provides, and get the status of the first child.

  The reason for this is that the second child has now become an “orphan” (his parent, the first child, has died and was assembled by your original process). As an orphan, it is passed to the proxy parent (in particular, “init”), which is always wait ing and, therefore, immediately collects all the zombies.

• In addition to the double fork, you can make your subprocesses in a separate session and / or refuse to control access to the terminal ("daemonize", in Unix-y terms). (This is a bit dirty and OS-dependent, I encoded it before, but for some corporate code, I don’t have access now.)

• Finally, you can simply periodically collect these processes. If you are using the subprocess module, just call the .poll function for each process, when convenient. This will return None if the process is still running, and completion status (after building it) if it is finished. If some of them still work, your main program may exit anyway while they continue to work; at this moment they become orphans, as in method No. 2 above.

The "ignore SIGCHLD" method is simple and simple, but has the disadvantage of interfering with library procedures that create and wait for subprocesses. There is work in Python 2.7 and later ( http://bugs.python.org/issue15756 ), but this means that the library routines do not see any failures in these subprocesses.

[Edit: http://bugs.python.org/issue1731717 for p.wait() , where p is the process from subprocess.Popen ; 15756 specifically for p.poll() ; but in any case, if you do not have corrections, you need to resort to methods 2, 3 or 4.]

Answer 2

Take a look at http://docs.python.org/2/library/multiprocessing.html

It provides an API that is very similar to threads. You can wait for the child process to complete if you wish.

Answer 3

Ways torek are fine!

I found another way to handle a non-existent process;

we can use waitpid to redistribute a nonexistent process as necessary:

 import os, subprocess, time def recycle_pid(): while True: try: pid, status, _ = os.wait3(os.WNOHANG) if pid == 0: break print("----- child %d terminated with status: %d" %(pid, status)) except OSError,e: break print("+++++ start pid:", subprocess.Popen("ls").pid) recycle_pid() print("+++++ start pid:", subprocess.Popen("ls").pid) recycle_pid() time.sleep(1) recycle_pid() 

recycle_pid does not block, it can cause if necessary.

Answer 4

After the process is completed or destroyed, the operating system expects the parent process to collect the status of the child process. You can use the process () method to get the status:

 p = subprocess.Popen( ... ) p.terminate() p.communicate() 

Note that terminating a process allows the process to intercept the termination signal and do whatever it wants to do with it. This is important because p.communicate () is a blocking call.

If you do not want this behavior to use p.kill () instead of p.terminate (), which allows the process not to intercept the signal.

If you want to use p.terminate () and make sure the process is finished, you can use the psutil module to check the status of the process.