The most likely culprit is that Global Interpreter Lock (GIL) is not held by a thread when calling Python code, resulting in undefined mode. Check all paths that make direct or indirect Python calls, acquire a GIL before calling Python code.
GIL is a mutex around the CPython interpreter. This mutex prevents concurrent operations on Python objects. Thus, at any given time, a maximum of one thread, the one that acquired the GIL, is allowed to perform operations on Python objects. When multiple threads are present, calling Python code without holding the GIL results in undefined behavior.
C or C ++ threads are sometimes referred to as foreign threads in the Python documentation. The Python interpreter does not have the ability to control someone else's thread. Therefore, foreign threads are responsible for managing the GIL to allow parallel or parallel execution with Python threads. It is necessary to carefully consider:
- Stack debugging because Boost.Python may throw an exception.
- Indirect calls in Python, such as copy constructors or destructors
One solution is to wrap Python callbacks with a custom type that knows about GIL management.
Using the RAII class to control the GIL provides an elegant, secure solution. For example, with the following with_gil class, when a with_gil object is with_gil , the calling thread receives a GIL. When the with_gil object with_gil destroyed, it restores the GIL state.
And its use:
{ with_gil gil;
With the ability to control the GIL via with_gil , the next step is to create a functor that correctly controls the GIL. The following py_callable class will wrap boost::python::object and get a GIL for all the paths in which Python code is called:
By controlling boost::python::object in free space, you can freely copy shared_ptr without having to hold the GIL. This allows us to safely use the default constructor instance, assignment operator, destructor, etc.
You can use py_callable as follows:
Here is a complete example demonstration in which the Python extension calls a Python object as a callback from a C ++ stream:
#include <memory> // std::shared_ptr #include <thread> // std::this_thread, std::thread #include <utility> // std::forward #include <boost/python.hpp> /// @brief Guard that will acquire the GIL upon construction, and /// restore its state upon destruction. class with_gil { public: with_gil() { state_ = PyGILState_Ensure(); } ~with_gil() { PyGILState_Release(state_); } with_gil(const with_gil&) = delete; with_gil& operator=(const with_gil&) = delete; private: PyGILState_STATE state_; }; /// @brief Helper type that will manage the GIL for a python callback. /// /// @detail GIL management: /// * Acquire the GIL when copying the `boost::python` object /// * The newly constructed `python::object` will be managed /// by a `shared_ptr`. Thus, it may be copied without owning /// the GIL. However, a custom deleter will acquire the /// GIL during deletion /// * When `py_callable` is invoked (operator()), it will acquire /// the GIL then delegate to the managed `python::object` class py_callable { public: /// @brief Constructor that assumes the caller has the GIL locked. py_callable(const boost::python::object& object) { with_gil gil; object_.reset( // GIL locked, so it is safe to copy. new boost::python::object{object}, // Use a custom deleter to hold GIL when the object is deleted. [](boost::python::object* object) { with_gil gil; delete object; }); } // Use default copy-constructor and assignment-operator. py_callable(const py_callable&) = default; py_callable& operator=(const py_callable&) = default; template <typename ...Args> void operator()(Args... args) { // Lock the GIL as the python object is going to be invoked. with_gil gil; (*object_)(std::forward<Args>(args)...); } private: std::shared_ptr<boost::python::object> object_; }; BOOST_PYTHON_MODULE(example) { // Force the GIL to be created and initialized. The current caller will // own the GIL. PyEval_InitThreads(); namespace python = boost::python; python::def("call_later", +[](int delay, python::object object) { // Create a thread that will invoke the callback. std::thread thread(+[](int delay, py_callable callback) { std::this_thread::sleep_for(std::chrono::seconds(delay)); callback("spam"); }, delay, py_callable{object}); // Detach from the thread, allowing caller to return. thread.detach(); }); }
Interactive use:
>>> import time >>> import example >>> def shout(message): ... print message.upper() ... >>> example.call_later(1, shout) >>> print "sleeping"; time.sleep(3); print "done sleeping" sleeping SPAM done sleeping