How does Python block signals during os.system ("sleep ...")?

Question

When I run this Python script with os.system in Ubuntu 12.04:

 import os, signal signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n')) print 'status=%r' % os.system('sleep 5') 

and then I send SIGABRT to the script process many times over 5 seconds, I get the following output:

 status=0 HANDLER 

This means that signal delivery was blocked until sleep 5 , and then only one signal was delivered.

However, with subprocess.call :

 import os, signal, subprocess signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n')) print 'cstatus=%r' % subprocess.call('sleep 5', shell=True) 

All individual signals are delivered earlier:

 HANDLER HANDLER HANDLER cstatus=0 

To distinguish magic from glibc from magic in Python, I rewrote the Python script in C, so os.system became a system (3):

 #include <errno.h> #include <signal.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> static void handler(int signum) { (void)signum; write(2, "HANDLER\n", 8); } int main(int argc, char **argv) { int got; struct sigaction sa; (void)argc; (void)argv; memset(&sa, 0, sizeof sa); sa.sa_handler = handler; if (0 != sigaction(SIGABRT, &sa, NULL)) return 3; got = system("sleep 5"); return !printf("system=0x%x\n", got); } 

Signals received ahead of schedule:

 HANDLER HANDLER HANDLER system=0x0 

So, I realized that the magic is in Python 2.7, and not in eglibc. But where is the magic? Based on strace's output and looking at the posix_system function in Modules/posixmodule.c , I could not figure out how Python blocks the signal until os.system returns.

Relevant code from Modules/posixmodule.c :

 static PyObject *posix_system(PyObject *self, PyObject *args) { char *command; long sts; if (!PyArg_ParseTuple(args, "s:system", &command)) return NULL; Py_BEGIN_ALLOW_THREADS sts = system(command); Py_END_ALLOW_THREADS return PyInt_FromLong(sts); } 

Perhaps the magic is in Py_BEGIN_ALLOW_THREADS ?

Do I understand correctly that it is not possible for my Python signal handler (configured using signal.signal ) to return to os.system ?

Is it because the signal handlers are blocked (at the Python level, not at the OS level) until Py_END_ALLOW_THREADS returns?

Here is the output of strace Python code from os.system : http://pastebin.com/Wjn9KBye

Answer 1

Perhaps the magic is in PY_BEGIN_ALLOW_THREADS?

The magic is mainly in the system . system cannot return an EINTR, so the libc implementation aims to resume its wait IN for the child process. This means that when using os.system control never returns to python until the underlying system completes, and therefore the python signal processing mechanics are not called in a timely manner.

subprocess.call , however, basically does this:

 # Compare subprocess.py:Popen/_eintr_retry_call(os.waitpid, self.pid, 0) while True: try: return os.waitpid(the_child_pid, 0) except OSError, e: if e.errno == errno.EINTR: # signal.signal() handler already invoked continue raise 

Here the control returns python when the main wait is interrupted. OSError / EINTR queries python to see if any signals have been triggered and, if so, to call the user-called code block associated with that signal. (And the way the interpreter adapts the semantics of the system signal: set the flag and check it between the "atomic" operations of python, calling the user code if necessary.)