ANSWERED
Ok, I solved this problem. All of this is how you initialize the state of the stream. You do not need to use ReleaseLock at all. Just add InitThreads to define your module:
BOOST_PYTHON_MODULE(ModuleName) { PyEval_InitThreads(); ... }
Well, I tried to diagnose this problem for hours and poured through what seems like every example on the Internet. Get tired now, so I can skip something obvious, but here's what happens:
I am wrapping a library in boost python. I run a python script that imports lib, constructs some objects, and then receives callbacks from C ++, which calls back to python. Before I call any python functions, I will try to get the global lock of the interpreter. Here is a sample code:
class ScopedGILRelease { public: inline ScopedGILRelease() { d_gstate = PyGILState_Ensure(); } inline ~ScopedGILRelease() { PyGILState_Release(d_gstate); } private: PyGILState_STATE d_gstate; }; class PyTarget : public DingoClient::ClientRequest::Target, public wrapper<DingoClient::ClientRequest::Target> { public: PyTarget(PyObject* self_) : self(self_) {} ~PyTarget() { ScopedGILRelease gil_lock; } PyObject* self; void onData(const boost::shared_ptr<Datum>::P & data, const void * closure) { ScopedGILRelease gil_lock;
The onData method of the Target object is called by the library as a callback. In python, we inherit PyTarget and implement another method. Then we use call_method <> to call this method. gil_lock receives a lock and through RIAA guarantees that the received state of the stream is always one version and that in fact it is always issued when leaving the scope.
However, when I run this in a script that tries to get a large number of callbacks for this function, it always segfaults. The script looks something like this:
In addition, the python script always creates an object that works:
PyEval_InitThreads(); PyEval_ReleaseLock();
before receiving any callbacks.
I have reduced the code to the place where I do not even call python in onData, I just acquire a lock. On release, it always crashes with:
Fatal Python error: ceval: tstate mix-up Fatal Python error: This thread state must be current when releasing
or
Fatal Python error: ceval: orphan tstate Fatal Python error: This thread state must be current when releasing
It seems random. I'm crazy here because I feel I am using GIL lock correctly, but it doesn't seem to work at all.
Other notes: Only one thread ever calls the Target object onData method.
When I sleep in a while loop in the python calling module with time.sleep (), it looks like the script will run longer, but eventually the script will perform similar problems. The time it lasts is proportional to the amount of time.sleep (i.e. time.sleep (10) lasts longer than time.sleep (0.01). It makes me think about how the script re-acquires GIL without my permission .
PyGILState_Release and PyGILState_Ensure are not called anywhere else in my code, no, where else do you need to call in python.
Update
I read another question that suggests importing threads in a module as an alternative to running
PyEval_InitThreads(); PyEval_ReleaseLock();
However, it does not work when I import threads in front of my module and remove the above two lines from my boost python shell.