Why is my thread / multiprocessing python script not working correctly?

Question

I have a script server that should be able to disable. When testing regular try..except I realized that Ctrl-C does not work in the usual way. I usually complete long tasks like

 try: ... except KeyboardInterrupt: #close the script cleanly here 

therefore, the task can be disabled on Ctrl-C . I have never encountered any problems with this before, but for some reason, when I pressed Ctrl-C , when this particular script works, the script just exits without catching Ctrl-C .

The initial version was implemented using Process from multiprocessing . I rewrote the script using Thread from threading , but in the same place. I used threading many times before, but I'm new to the multiprocessing library. In any case, I have never experienced this Ctrl-C behavior before.

Usually, I always performed the functions of sentries, etc., to close the Queues and Thread instances in an orderly manner, but this script just comes out without an answer.

Finally, I tried overriding signal.SIGINT just like this

 def handler(signal, frame): print 'Ctrl+C' signal.signal(signal.SIGINT, handler) ... 

Here Ctrl+C was actually caught, but the handler does not execute, it never prints anything.

Besides the threading / multiprocessing aspect, parts of the script contain C++ SWIG objects. I do not know if this is related to this. I am running Python 2.7.2 on OS X Lion.

So a few questions:

• What's going on here?
• How can I debug this?
• What do I need to know to understand the root cause?

PLEASE NOTE: The internal elements of a script are property, so I cannot provide code examples. However, I really want to get pointers to debug this myself. I am experienced enough to understand this if someone can point me in the right direction.

EDIT: I started commenting on imports, etc., to find out what caused the strange behavior, and I narrowed it down to importing the C++ SWIG . Any ideas why importing a C++ SWIG intercepts Ctrl-C ? However, I am not the author of the guilty library, and my SWIG experience is limited, so I don’t know where to start ...

EDIT 2: I just tried the same script on a Windows machine, and in Windows 7 Ctrl-C caught as expected. I am not going to worry about the OS X part, the script will work on Windows anyway.

Answer 1

This may be due to the way Python manages C flows, signals, and calls.

In short - Ctrl-C cannot interrupt calls to C, because the implementation requires that the python thread process the signal, not just some thread, but the main thread (often blocked, waiting for other threads).

In fact, long operations can block everything.

Consider this:

 >>> nums = xrange(100000000) >>> -1 in nums False (after ~ 6.6 seconds) >>> 

Now try to press Ctrl-C (without interruption!)

 >>> nums = xrange(100000000) >>> -1 in nums ^C^C^C (nothing happens, long pause) ... KeyboardInterrupt >>> 

The reason Ctrl-C doesn’t work with streaming programs is because the main thread is often blocked during a continuous streaming connection or blocking (for example, any “wait”, “connection” or just an empty empty “main” thread, which in background causes python to "join" to any child thread).

Try to insert simple

 while True: time.sleep(1) 

in the main topic.

If you have a long C function, perform signal processing at level C (let the Force be with you!).

It is mainly based on a David Bezlie video on this subject.

Answer 2

It exits because something else is more likely to catch KeyboardInterupt, and then raises some other exception, or simply returns None. You should get a trace for debugging. You need to capture the output of stderr or run the script with the -i command-line option so you can see the trace. Also, add another block to catch all other exceptions.

If you suspect that a C ++ function call should catch CTRL + C, try outputting it. If the C function returns nothing, you can do nothing but ask the author to add some exception handling, return codes, etc.

 try: #Doing something proprietary ... #catch the function call output result = yourCFuncCall() #raise an exception if it not what you expected if result is None: raise ValueError('Unexpected Result') except KeyboardInterupt: print('Must be a CTRL+C') return except: print('Unhandled Exception') raise 

Answer 3

how about atexit? http://docs.python.org/library/atexit.html#module-atexit