How to use Py_AddPendingCall

Question

I have a built-in Python program that runs in a stream in C.

When the Python interpreter switches the context of the thread (giving control to another thread), I would like to receive a notification in order to perform some necessary operations.

Py_AddPendingCall seems to be exactly what I'm looking for. However, the API docs are pretty brief about this function, and I'm confused about how it is supposed to use Py_AddPendingCall . I am reading documents, I understand that:

• The worker thread calls Py_AddPendingCall and the handler assigns the function.
• When the Python interpreter starts, it calls the handler function when it gives control to another thread.
• The handler itself is executed in the main interpreter thread, with GIL acquired

I googled around, for example, code showing how to use Py_AddPendingCall , but I can not find anything. My own attempt to use it just doesn't work. The handler just does not get called.

My working thread code:

 #include <pthread.h> #include <stdlib.h> #include <stdio.h> const char* PYTHON_CODE = "while True:\n" " for i in range(0,10): print(i)\n" "\n"; int handler(void* arg) { printf("Pending Call invoked!\n"); abort(); } void* worker_thread(void*) { PyGILState_STATE state = PyGILState_Ensure(); int res = Py_AddPendingCall(&func, nullptr); cout << "Result: " << res << endl; PyRun_SimpleString(CODE); PyGILState_Release(state); return 0; } int main() { Py_Initialize(); PyEval_InitThreads(); PyEval_ReleaseLock(); pthread_t threads[4]; for (int i = 0; i < 4; ++i) pthread_create(&threads[i], 0, worker_thread, 0); for (int i = 0; i < 4; ++i) pthread_join(threads[i], 0); Py_Finalize(); } 

In this example, worker_thread is called in C as the pthread worker thread. In this test, I run 4 worker threads, so some context switching should happen. This loop is infinite, but the pending call handler is never called.

So, can anyone provide a minimal working example that shows how to use Py_AddPendingCall ?

Answer 1

Pending calls are only made in the main thread, and the main thread can only serve pending calls when it executes Python code. Thus, the main thread must start the interpreter loop to serve any pending calls. From Python/ceval.c :

 Py_MakePendingCalls(void) { ... /* only service pending calls on main thread */ if (main_thread && PyThread_get_thread_ident() != main_thread) return 0; ... } 

In the deployment scenario, “main thread” refers to any thread that calls PyEval_InitThreads (note that it also PyEval_InitThreads called automatically the first time the thread is run from Python code). Since your main thread is in pthread_join , it does not execute Python code and therefore does not send pending calls.

Please note that the handler is executed only when control is transferred to the main thread - the handler does not start if control passes from one worker to another. Therefore, Py_MakePendingCalls is probably not the right interface to use for this task.