Python can import a module that is not installed

Question

So, I play with the python wrapping script I wrote, and it has a submodule, let it submodule it. The folder structure is as follows:

 cool_script/ setup.py cool_script.py submodule/ __init__.py implementation.py 

Now, after many calls to pip install . and pip install -e . , I have a situation where a submodule can be imported globally. No matter where on my system this will always work:

 $ python3 [...] >>> import submodule >>> submodule.__file__ '/home/me/fake/path/cool_script/submodule/__init__.py' 

But I do not know why.

The package I made was removed again, and pip could not find the submodule in its index. There is nothing in the distributions, I manually deleted cool_script.egg-link , which was still sitting there:

 $ ls /usr/local/lib/python3.4/dist-packages | ack cool $ ls /usr/local/lib/python3.4/dist-packages | ack submodule $ 

PYTHONPATH also empty:

 $ echo $PYTHONPATH $ 

Why does Python know the location of the submodule ? How can I find out?

Answer 1

First run python -c "import site; print(site.getsitepackages())" . It will print a list like this:

 ['/XXX/something/site-packages'] 

Usually there is only one path to this list, and it points to the directory where pip installs your scripts. You can ls into it if you're interested: ls /XXX/something/site-packages/ .

More interestingly, however, pip places the "link" file in this directory when you use the developer installation (aka pip install -e ). The "link" file is named after the original project with the extension .egg-link at the end.

So you probably have the cool_script.egg-link file in this directory. And if you try to print it, you should find that its contents reflect the original location of the file system of your module. Sort of:

 $ cat /XXX/something/site-packages/cool_script.egg-link /home/me/fake/path/cool_script/ . 

This is how pip writes that it installed something in developer mode, but not in the way Python really knows how to find your module (it would be too easy, right ?: -)).

Python does not know about .egg-link files, but it reads all .pth files in the site-packages directory to get additional paths for sys.path (*). Thus, in order for Python to import developer mode settings, pip writes all its paths to a single .pth file, usually called easy-install.pth (because the old easy-install tool actually used this technique for the first time). And if you print this file, you will get a list of all the project paths installed in developer mode:

 $ cat /XXX/something/site-packages/easy-install.pth /home/me/fake/path/cool_script/ /home/me/another/project/ 

And you can check if all those paths listed in easy-install.pth are really added to your sys.path .

(*) Technically, the part of Python that reads these .pth files is a site module, which is usually imported automatically at startup. There is an option to disable the site module, although, for example, using python -S . In this case, you will see that sys.path contains neither the site-packages directory, nor the developer installation path.