The library expects a character in a flat namespace, although compiled with a two-level namespace

Question

I dynamically load Python using dlopen and RTLD_LOCAL to avoid collisions with another library, which coincidentally contains several characters with the same name. Running my MVCE above on macOS with an Xcode error because it expects _PyBuffer_Type in the global namespace.

 Traceback (most recent call last): File "...lib/python2.7/ctypes/__init__.py", line 10, in <module> from _ctypes import Union, Structure, Array ImportError: dlopen(...lib/python2.7/lib-dynload/_ctypes.so, 2): Symbol not found: _PyBuffer_Type Referenced from: ...lib/python2.7/lib-dynload/_ctypes.so Expected in: flat namespace in ...lib/python2.7/lib-dynload/_ctypes.so Program ended with exit code: 255 

But why? Does RTLD_LOCAL two-level namespace?

I used otool -hV to verify that _ctypes.so was compiled using the Two-Level namespace option. In my understanding, to solve a symbol, the library name + the name of the symbol itself is required. Why does he expect _PyBuffer_Type in a flat namespace and / or why can't he find it? See TWOLEVEL to the right

 > otool -hV /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 X86_64 ALL 0x00 BUNDLE 14 1536 NOUNDEFS DYLDLINK TWOLEVEL 

Any idea what is going on here?

MVCE

You can copy to a new Xcode project, just compile and execute.

 #include </System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/Python.h> #include <dlfcn.h> int main(int argc, const char * argv[]) { auto* dl = dlopen("/System/Library/Frameworks/Python.framework/Versions/2.7/Python", RTLD_LOCAL | RTLD_NOW); if (dl == nullptr) return 0; // Load is just a macro to hide dlsym(..) #define Load(name) ((decltype(::name)*)dlsym(dl, # name)) Load(Py_SetPythonHome)("/System/Library/Frameworks/Python.framework/Versions/2.7"); Load(Py_Initialize)(); auto* readline = Load(PyImport_ImportModule)("ctypes"); if (readline == nullptr) { Load(PyErr_Print)(); dlclose(dl); return -1; } Py_DECREF(readline); Load(Py_Finalize)(); return 0; } 

Answer 1

This question and your related RTLD_GLOBAL question concern the semantics of a dynamic loader allowing undefined characters in shared libraries that it loads. I was hoping to find an explicit link to the documentation that would explain what you see, but I could not do it. However, I can make an observation that can explain what is happening.

If we run verbosity, we will see that the python library is trying to load two shared libraries before the failure:

 bash-3.2$ PYTHONVERBOSE=1 ./main 2>&1 | grep -i dlopen dlopen(".../python2.7/lib-dynload/_locale.so", 2); dlopen(".../python2.7/lib-dynload/_ctypes.so", 2); 

Given that the former succeeds, we know that usually the dynamic loader resolves undefined characters against the namespace of the calling library. And actually, as you noticed in the comments on your other question, this even works when there are two versions of the python library, i.e. dlopen() executed by python libraries according to their respective namespaces. So far it sounds the way you want. But why is _ctypes.so not loading?

We know that _PyModule_GetDict is the symbol that _locale.so called so as not to load into your other question; and that he obviously works here. We also know that the _PyBuffer_Type character _PyBuffer_Type not work here. What is the difference between these two characters? Search them in python library:

 bash-3.2$ nm libpython2.7.dylib | grep _PyModule_GetDict 00000000000502c0 T _PyModule_GetDict bash-3.2$ nm libpython2.7.dylib | grep _PyBuffer_Type 0000000000154f90 D _PyBuffer_Type 

_PyModule_GetDict is a Text (code) character, while _PyBuffer_Type is a Data character.

Therefore, based on this empirical data, I suspect that the dynamic loader will allow undefined characters against RTLD_LOCAL code characters of the calling library, but not RTLD_LOCAL data characters. Perhaps someone may point to an explicit link.