Python 3.6.0 implicit namespace package

Question

I found strange behavior with an implicit namespace package in Python 3.6.0rc1. Could you tell me if I am wrong or is this a Python 3.6 bug?

I am working with the marrow namespace package, which has two separate packages, marrow.util and marrow.mailer . The second depends on the first.

Suppose we installed marrow.util in site-packages for Python 2.7, 3.5 and 3.6:

 $ ls -la /usr/lib/python*/site-packages/marrow /usr/lib/python2.7/site-packages/marrow: total 24 drwxr-xr-x. 3 root root 4096 Dec 23 12:23 . drwxr-xr-x. 196 root root 16384 Dec 23 12:23 .. drwxr-xr-x. 3 root root 4096 Dec 23 12:23 util /usr/lib/python3.5/site-packages/marrow: total 12 drwxr-xr-x. 3 root root 4096 Dec 23 12:24 . drwxr-xr-x. 99 root root 4096 Dec 23 12:24 .. drwxr-xr-x. 4 root root 4096 Dec 23 12:24 util /usr/lib/python3.6/site-packages/marrow: total 12 drwxr-xr-x. 3 root root 4096 Dec 23 14:25 . drwxr-xr-x. 37 root root 4096 Dec 23 14:25 .. drwxr-xr-x. 4 root root 4096 Dec 23 14:25 util 

There are no __init__.py files here, which is true because marrow is a namespace package. This log message can be seen during installation:

 Skipping installation of <deleted>/site-packages/marrow/__init__.py (namespace package) 

And then you have the second part of the marrow marrow.mailer namespace package built-in (but not installed) in another directory. For example, for example:

 $ pwd /builddir/build/BUILD/marrow.mailer-4.0.2 $ ls coverage.xml debuglinks.list elfbins.list LICENSE.txt marrow.mailer.egg-info README.textile setup.py debugfiles.list debugsources.list example marrow PKG-INFO setup.cfg test $ ls marrow/ __init__.py __init__.pyc mailer __pycache__ 

When I run Python 2.7.12 or 3.5.2 in this folder and try to import marrow.util (from the site packages), it works as expected.

 $ pwd /builddir/build/BUILD/marrow.mailer-4.0.2 $ python2 Python 2.7.12 (default, Sep 29 2016, 12:52:02) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import marrow.util >>> $ python3.5 Python 3.5.2 (default, Sep 14 2016, 11:28:32) [GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import marrow.util >>> 

But when I try to import the same module with Python 3.6, it fails:

 $ python3.6 Python 3.6.0rc1 (default, Dec 10 2016, 14:50:33) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import marrow.util Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'marrow.util' >>> 

I found this problem when I tried to build marrow.mailer as an RPM package in Mock. Everything works with Python 2.7 and 3.5, but Python 3.6 cannot import marrow.util from site packages, and for this reason the marrow.mailer tests fail during RPM build.

Trace example from failed tests:

 Traceback: test/test_addresses.py:8: in <module> from marrow.mailer.address import Address, AddressList, AutoConverter marrow/mailer/__init__.py:12: in <module> from marrow.mailer.message import Message marrow/mailer/message.py:21: in <module> from marrow.mailer.address import Address, AddressList, AutoConverter marrow/mailer/address.py:12: in <module> from marrow.util.compat import basestring, unicode, unicodestr, native E ModuleNotFoundError: No module named 'marrow.util' 

I cannot find anything related to this problem in Changelog for Python 3.6.

Thanks for any help.

EDIT: I checked sys.path in Python 3.6 and everything looks fine:

 $ python3.6 Python 3.6.0rc1 (default, Dec 10 2016, 14:50:33) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages'] >>> 

EDIT 2:

Since I still cannot find any solution, and I have no answer, I created a simple Bash script that can reproduce my situation. The only thing you need is Python 3.5 and Python 3.6.

 #!/bin/bash # Change this to run script with different Python #PYTHON=python3.5 # system Python 3.5 PYTHON=~/temp/Python-3.6.0/python # compiled Python 3.6 # Create venv and activate $PYTHON -m venv venv source ./venv/bin/activate # Install marrow.util package as a part of namespace package marrow pip install marrow.util # Create simple folder structure mkdir -p marrow/mailer # Create structure of __init__.py files # For namespace package with related content cat >> marrow/__init__.py << EOL try: # pragma: no cover __import__('pkg_resources').declare_namespace(__name__) except ImportError: # pragma: no cover __import__('pkgutil').extend_path(__path__, __name__) EOL # For mailer module just with print() cat >> marrow/mailer/__init__.py << EOL print('Imported!!!') EOL # Testing # Importing marrow.util installed via pip in venv $PYTHON -c "import marrow.util" # Importing marrow.mailer created manually in PWD $PYTHON -c "import marrow.mailer" # deactivate venv deactivate 

If you run this script with Python 3.5, you will see that Python 3.5 can import marrow.util installed via pip, but cannot import marrow.mailer into a local folder. But Python 3.6 can import the local marrow.mailer module and cannot import the marrow.util module.

Answer 1

These packages do not use an implicit namespace ("native namespaces") or, if you have versions that are, pin your dependencies to make sure that you are not mixing the old and new style namespace. These are completely incompatible approaches.

It looks like you are trying in your MCVE code example to create a namespace package ( marrow/__init__.py announces through the old trick of explicitly replacing Python 2 declarations), AKA pkg-resources-style namespace packages . This requires an argument for setup.py (actual packaging) and setting metadata through package installation. In particular, this method includes the tricks of the .pth file (see $VIRTUAL_ENV/lib/python3.?/site-packages ) if it is installed "in development", and extracting / unpacking to this installation path. Without this, there really will not be a namespace, and your code will not be found, not with this old style. (The first one set will win.)

In REPL, you can import a namespace, for example, import marrow , and then examine marrow.__path__ to see what was found / included as a diagnostic tool; my current WIP virtual environment on this machine has m.package , m.schema and m.interface , which makes sense since I recently m.interface . A more modern approach to your own namespace allows for more arbitrary blending. The idea is that a folder without __init__.py simply a namespace automatically concatenated through PYTHONPATH , but alas, this is not how namespaces are used to work. (All participants need this __init__.py stub without any other code at all levels of the namespace.)

I am in the process of modernizing the entire Marrow ecosystem (I already started with a few, as noted above), to eliminate the legacy of Python 2 and start using new Python 3 structures and approaches, including the modern namespace. The main version bug for everything and dependencies should be fixed less than these versions for any code that still needs old namespaces or in Python 2.

(I re-request local Python 3.6 and 3.5 for further study.)