RTLD_GLOBAL and two-level namespaces on macOS

Question

After reading the Apple documentation for Executing Mach-O Files, it says:

The two-level namespace function of OS X v10.1 and later adds the module name as part of the symbol name of the characters defined in It. This approach ensures that module symbol names do not conflict with names used in other modules .

So in my example, I am loading python2 and python3 into the same process. Both Python libraries (by default) are compiled with a two-level version of the namespace. Both libs are also loaded with the RTLD_GLOBAL flag via dlopen (..), so characters with the same name should not interfere with each other, since the two modules have different names (python27 and python36).

Example:

 #include <{...}/include/python2.7/Python.h> int main(int argc, const char * argv[]) { auto* py3 = dlopen(".../python36", RTLD_GLOBAL | RTLD_NOW); if (py3 == nullptr) return 0; auto* py2 = dlopen(".../python27", RTLD_GLOBAL | RTLD_NOW); if (py2 == nullptr) return 0; auto* init = ((decltype(Py_Initialize)*)dlsym(py2, "Py_Initialize")); if (init) { init(); } return 0; } 

The problem is that after importing python2 /path/to/python2/lib/lib-dynload/_locale.so , the PyModule_GetDict function is PyModule_GetDict from python3. Why is this? How can this happen? Shouldn't this hinder a two-level namespace?

PS lib-dynload is a directory with additional C modules for Python on macOS. I have verified that the correct _local.so lib is _local.so from the python2 environment.

Edit:

After some experimentation, I saw that the symbols of the first loaded python lib always get a higher priority, but are not sure, although if it is intended for the first loaded libs or still "undefined ground behavior".

Call Py_Initialize () from python27 - Success:

 1. Loading python27 first 2. Loading python36 second 3. PYTHONHOME to python27 4. cal Py_Initialize() of python27 

Py_Initialize () call python27 - Failure:

 1. Loading python36 first 2. Loading python27 second 3. PYTHONHOME to python27 4. cal Py_Initialize() of python27 

I get the same results differently.

Call Py_Initialize () for python36 - Success:

 1. Loading python36 first 2. Loading python27 second 3. PYTHONHOME to python36 4. cal Py_Initialize() of python36 

Py_Initialize () call python36 - Failure:

 1. Loading python27 first 2. Loading python36 second 3. PYTHONHOME to python36 4. cal Py_Initialize() of python36 

Answer 1

Symbols in libpython (e.g. libpython2.7.dylib) resolve correctly. For example, in the scenario described above, I see that PyModule_GetDict() receives a call 155 times before an incorrectly allowed call.

The problem is that python itself compiles shared libraries and uses dlopen() to load them. You can see the dlopen() event by dlopen() environment PYTHONVERBOSE at startup:

 $ PYTHONVERBOSE=1 ./main 2>&1 | grep dlopen 

which produces:

 dlopen(".../lib/python2.7/lib-dynload/_locale.so", 2); 

Argument 2 corresponds to RTL_NOW , but this does not really matter. The problem is that this separate library cannot indicate that its characters should be allowed in the libpython2.7.dylib library. However, it has several python characters; in particular the one that causes the problem:

 $ nm prefix/lib/python2.7/lib-dynload/_locale.so | grep GetDict U _PyModule_GetDict 

So, when the python dlopen() library, all it can do is resolve a character without qualification. Apparently, the semantics of dl functionality is the resolution of such characters based on the loading order of the libraries, as you noted.

So everything works fine until we load _locale.so , as you can see from the following backtrace:

 * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x50) * frame #0: 0x00000001003f3fc1 libpython3.6m.dylib`PyErr_FormatV [inlined] PyErr_Restore at errors.c:42 [opt] frame #1: 0x00000001003f3fb7 libpython3.6m.dylib`PyErr_FormatV [inlined] PyErr_Clear at errors.c:355 [opt] frame #2: 0x00000001003f3fb7 libpython3.6m.dylib`PyErr_FormatV(exception=0x00000001004cba18, format="%s:%d: bad argument to internal function", vargs=0x00007fff5fbfdcb0) at errors.c:841 [opt] frame #3: 0x00000001003f2c39 libpython3.6m.dylib`PyErr_Format(exception=<unavailable>, format=<unavailable>) at errors.c:860 [opt] frame #4: 0x0000000100358220 libpython3.6m.dylib`PyModule_GetDict(m=0x0000000101a5a868) at moduleobject.c:450 [opt] frame #5: 0x00000001000f491c _locale.so`init_locale at _localemodule.c:703 [opt] frame #6: 0x00000001018d1176 libpython2.7.dylib`_PyImport_LoadDynamicModule(name="_locale", pathname=".../lib/python2.7/lib-dynload/_locale.so", fp=<unavailable>) at importdl.c:53 [opt] 

It is also worth noting that _locale.so is just the first library that fails. If you somehow walked past it, there are quite a few other libraries that would potentially have similar problems in .../lib/python2.7/lib-dynload .