An import structure that works both in packages and externally, in both Python 2 and 3?

Question

When I developed the package exclusively for Python 2, I could use the simple import b syntax to import the relative path without worrying about whether the importing file was in the package or not. This had the advantage that I could run the if __name__ == "__main__": block if __name__ == "__main__": any file, just by executing the file, and all the imported files would work fine.

After adding Python 3 support, I had to move on to the new relative import syntax, which also supports 2.7: from . import b from . import b . However, this syntax only works inside packages. Direct file execution does not directly work:

 Traceback (most recent call last): File "./a.py", line 2, in <module> from . import b ValueError: Attempted relative import in non-package 

The workaround is to invoke the file by importing it as a module from the top directory:

 python -m foo.a 

However, this places the requirement in the working directory, which prevents you from transferring the output to another program that also takes care of the working directory.

Is there any way to get a cake and eat it? That is, they support both running as a script and importing as part of the package, working both in Python 2 and 3?

---

Example package structure:

 foo/ foo/__init__.py foo/a.py (imports b) foo/b.py (imports c) foo/c.py 

I would like both of them to work for x in (a, b, c):

 import foo.x (in some file when foo/ is in path) python[23] path/to/foo/x.py 

---

The comment below mentions setting __package__ according to PEP 366 , but "if the script moves to another package or subpackage, the template panel must be manually updated."

Update . I tried to execute the PEP 366 solution, but could not understand. It says:

The additional code that sys.path manages will be required for direct execution to work without the top-level package already being imported.

This is the case when executing a file from an unimported package. What does this additional code look like?

Answer 1

Here is a solution based on KronoS answer and comments that allow you to use the same template for modules regardless of path or package name:

 if __name__ == "__main__" and __package__ == None: import importlib import os.path import sys def _gen_path(): head, tail = os.path.split(os.path.realpath(__file__)) while head: if not os.path.isfile(os.path.join(head, '__init__.py')): yield head return head, tail = os.path.split(head) yield tail def _load_package(): path = list(_gen_path()) syspath = sys.path[:] sys.path[:0] = [path.pop()] package = '.'.join(reversed(path)) importlib.import_module(package) sys.path = syspath return package __package__ = _load_package() 

It looks at the file path until the __init__.py tagging packages are marked, and then imports the parent module package, setting __package__ correctly. After that, comparative imports, such as from ..bar import baz , just work.

---

Including these functions too badly in their own module returns you to the square. In addition, there is no Python 2/3 portable way to limit the effects of sys.path changes to this import only, so everything in the base directory can obscure the absolute import in any module or package that is imported using the parent package.

Answer 2

Is there any way to get a cake and eat it? That is, they support both running as a script and importing as part of the package, working both in Python 2 and 3?

No ... Maybe ... But from what I can gather from what you are trying to do, you make the problem even more complex than it should be. I would create the package as normal, where you can support both Python 2 and 3. Then install using setup script and import the package into the script without using relative paths. This gives you the freedom to execute the script anywhere and the compatibility with Python 2 and 3 of your package.

---

I still adhere to my initial statement above, in which I believe that you are making it more complicated than it really should be, or you are not giving us all the information about why it should be done this way. However, if you follow what PEP 366 claims, this should work. In your modules, where the script is located (i.e. Contains if __name__ == "__main__": , add the following lines to the beginning (or to your main if __name__ == "__main__": file:

 if __name__ == "__main__" and __package__ == None: __package__ == "expected.package.name" sys.path.append(<path to root package 'expected'>) 

This, of course, means that you will have to manually update them if you ever move the script, or if the package moves, or anything related to this path moves (hence why I still think the installation is by far the best option here).

Answer 3

Well, you can add the directory where a and b PYTHONPATH modules live (ref: https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH ).

In addition, as indicated in the link above, the directory in which module a resides is automatically added to PYTHONPATH if a is the main module. For example, if you have the following code in the /test/a.py and /test/b.py :

/test/a.py

 if __name__ == '__main__': import sys print(sys.path) import b print('this is a') 

/test/b.py

 print('this is b') 

And you do a like:

 $ cd /test/ $ python3 a.py 

You will get the result:

 ['', '/usr/lib/python34.zip', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-linux', '/usr/lib/python3.4/lib-dynload', '/usr/lib/python3.4/site-packages'] this is b this is a 

In addition, if you do:

 $ python3 /test/a.py 

You will get the output:

 ['/test', '/usr/lib/python34.zip', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-linux', '/usr/lib/python3.4/lib-dynload', '/usr/lib/python3.4/site-packages'] this is b this is a