MPI Signal Processing

Question

When using, is mpirunit possible to catch signals (for example, SIGINT generated with ^C) in running code?

For example, I am running parallel python code. I can except KeyboardInterruptcatch these errors on startup python blah.pyby myself, but I can't do it mpirun -np 1 python blah.py.

Does anyone have a suggestion? Even finding how to catch signals in a compiled C or C ++ program would be a useful start.

If I send a signal to spawned Python processes, they can handle the signals correctly; however, signals sent to the parent process orterun(i.e. due to excess of time on the wall in the cluster or pressing the -C control key in the terminal) will immediately destroy everything.

Answer 1

If you use mpirun --nw, then mpirunit should be completed as soon as it starts the subprocesses, instead of waiting for them to complete; if this is acceptable, I believe that your processes will be able to catch their own signals.

Answer 2

I think it really depends on the implementation.

• In SLURM, I tried to use sbatch --signal USR1@30to send SIGUSR1(whose sign is 30.10 or 16) to a program launched by commands srun. And the process got a signal SIGUSR1 = 10.

• For IBM MPI platform, according to https://www.ibm.com/support/knowledgecenter/en/SSF4ZA_9.1.4/pmpi_guide/signal_propagation.html

SIGINT, SIGUSR1, SIGUSR2 will bypass processes.

• MPICH SIGUSR1 . ref: http://lists.mpich.org/pipermail/discuss/2014-October/003242.html >

• MPI SIGUSR1 SIGUSR2 mpiexec . ref: http://www.open-mpi.org/doc/v1.6/man1/mpirun.1.php#sect14 >

• IntelMPI, https://software.intel.com/en-us/mpi-developer-reference-linux-hydra-environment-variables

I_MPI_JOB_SIGNAL_PROPAGATION I_MPI_JOB_TIMEOUT_SIGNAL .

: python cython, SIGUSR1 , - .

Answer 3

signal signal.signal:

Install the signalnum signal handler for the function handler. the handler can be a Python object called with two arguments (see below) or one of the special values ​​signal.SIG_IGN or signal.SIG_DFL. The previous signal handler will be returned ...

import signal
def ignore(sig, stack):
  print "I'm ignoring signal %d" % (sig, )

signal.signal(signal.SIGINT, ignore)
while True: pass

If you send SIGINTthe Python interpreter with this script (via kill -INT <pid>), it will print the message and just continue execution.