All geek questions in one place

Pyth VM callback pthread

Let's say I have a python script that loads a shared library (SL) through ctypes.

  • SL installs a pthread T1
  • python script sets up callbacks through SL ie python script calls functions from SL with links to python calls

alt text http://www.gliffy.com/pubdoc/1993061/L.jpg

Now, let's say T1 calls the callback function, the following assumptions are true:

  • Python side callback function executes in T1 context
  • I can use queue to communicate between T1 and Python VM
  • I will need to poll the specified queue on the side of the Python virtual machine

I understand all the concepts of streaming processing, general state, etc., but I didn’t dig the multithreaded side with Python very deeply. Since there is a level of adaptation that I don’t know at the moment (ctypes), I’m afraid that some key aspects of the process will not be enough for me.

+4
source share
2 answers

Polling a queue is usually not required (you can devote another thread on the Python side to blocking .get calls on it), but this is not very important. The problem is that with such an agreement you can get to the GIL - see the Three links from this page on Wikipedia, where you can find many ways.

When you connect to / from C using C (or Cython ) code using the Python C API, you can free and acquire GIL quite simply, at least hopefully by avoiding deadlocks and the like; with ctypes, GIL operations are automated when you call back to / from C situations, so if there is any other lock in the game, a deadlock is a risk (since things are not under your control, you cannot easily provide Djikstra with the Banker algorithm applied ).

+1
source

This can be easily implemented using Cython. I wrote a sample at https://github.com/sachinpc/cython/tree/master/Demos/mt_callback

This example shows how to wrap a python callback in cython when called from pthread. For your specific use case, you can add to the queue or deactivate the object in the stream code from the callback function. In the same example, I added in the extension for both pthread-id and python-thread-id.

0
source

Source: https://habr.com/ru/post/1301451/

More articles:


All Articles