I get an error when I call the C_ API Py_Finalize () from a different C stream than I made a python call.
The error I see is:
Exception ignored in: <module 'threading' from 'C:\\Python34-32\\Lib\\threading.py'> Traceback (most recent call last): File "C:\Python34-32\Lib\threading.py", line 1289, in _shutdown assert tlock.locked() AssertionError:
This only happens in Python 3.X (tested since 3.4.2), in Python 2.7 the exact same code has no problems.
Here is a minimal example that shows that this happens when using a C-stream, but not when everything happens on the same c-stream:
#include <iostream> #include <fstream> #include <thread> #include <cassert> #include <Python.h> void make_file() { std::fstream file("my_test.py", std::ios::out); file << "import threading\n" << "def my_function():\n" << " pass\n" ; file.close(); } void exec() { PyGILState_STATE gstate = PyGILState_Ensure(); PyObject* pdict = PyDict_New(); PyDict_SetItemString(pdict, "__builtins__", PyEval_GetBuiltins()); PyRun_String("import my_test", Py_file_input, pdict, pdict); PyRun_String("my_test.my_function()", Py_file_input, pdict, pdict); assert(!PyErr_Occurred()); PyGILState_Release(gstate); } void basic() { std::cout << "--Starting Basic--" << std::endl; Py_Initialize(); PyEval_InitThreads(); PyThreadState* threadState = PyEval_SaveThread(); exec(); PyEval_RestoreThread(threadState); Py_Finalize(); std::cout << "--Basic Complete--" << std::endl; } void with_thread() { std::cout << "--Starting With Thread--" << std::endl; Py_Initialize(); PyEval_InitThreads(); PyThreadState* threadState = PyEval_SaveThread(); std::thread t(exec); t.join(); PyEval_RestoreThread(threadState); Py_Finalize(); std::cout << "--With Thread Complete--" << std::endl; } int main(int argc, char* argv[]) { make_file(); basic(); with_thread(); return 0; }
Output
--Starting Basic-- --Basic Complete-- --Starting With Thread-- Exception ignored in: <module 'threading' from 'C:\\Python34-32\\Lib\\threading.py'> Traceback (most recent call last): File "C:\Python34-32\Lib\threading.py", line 1289, in _shutdown assert tlock.locked() AssertionError: --With Thread Complete--
The order of the main calls () / with_thread () is basically irrelevant, I can even include these lines several times without affecting, each call to with_thread () produces an error.
Edit:
Make threadState global and then change exec to:
void exec() { //PyGILState_STATE gstate = PyGILState_Ensure(); PyEval_RestoreThread(threadState); PyObject* pdict = PyDict_New(); PyDict_SetItemString(pdict, "__builtins__", PyEval_GetBuiltins()); PyRun_String("import my_test", Py_file_input, pdict, pdict); PyRun_String("my_test.my_function()", Py_file_input, pdict, pdict); assert(!PyErr_Occurred()); //PyGILState_Release(gstate); threadState = PyEval_SaveThread(); }
makes the error go away, however then I have a global value that I need to coordinate between users of my library (in my actual code, the exec () function can be written by anyone, and I have a lot more initialization of the things that I run). Any ideas on how to make the GIL more isolated as a source example while maintaining thread compatibility?