A script call in the standard project directory structure (Python path for bin subdirectory)

Question

I am experimenting with the fact that my Python code is part of the standard directory structure used to deploy with setup.py and possibly PyPI. for a Python library called mylib, it would be something like this:

 mylibsrc/ README.rst setup.py bin/ some_script.py mylib/ __init.py__ foo.py 

Often there is also a subdirectory test/ , but I have not tried writing unit tests yet. The recommendation for scripting in the bin/ subdirectory can be found in the official Python packaging documentation.

Of course, scripts start with code that looks like this:

 #!/usr/bin/env python from mylib.foo import something something("bar") 

This works well when you end up deploying a script (e.g. devpi) and then installing it with pip. But if I ran the script directly from the source directory, as I would get this error when developing new changes in the / script library:

 ImportError: No module named 'mylib' 

This is true even if the current working directory is the root of mylibsrc/ , and I ran the script by typing ./bin/some_script.py . This is because Python begins to search for packages in the directory of the script to be started (i.e., from bin/ ), and not in the current working directory.

What is a good way to run scripts easily when developing packages?

Here is another relevant question (especially comments on the first answer).

The solutions for this that I have found so far fall into three categories, but none of them are perfect:

• Manually pin your Python module search path before running scripts.
  • You can manually add mylibsrc to my PYTHONPATH environment variable. This is apparently the most official (Pythonic?) Solution, but it means that every time I check a project, I have to remember to manually change my environment before I can run any code in it.
  • Add . to the beginning of my PYTHONPATH environment variable. As far as I understand, this may have some security issues. It would be my favorite trick if I were the only person who could use my code, but I do not, and I do not want to ask others to do it.
  • When browsing the Internet for files in the test/ directory, I saw recommendations that they all (indirectly) include a line of code sys.path.insert(0, os.path.abspath('..')) (for example, in structuring your project ). Ugh! This is similar to portable hacking for files that are intended only for testing, but not for those that will be installed with the package.
  • Edit: since then I have found an alternative that ends up in this category: by running scripts with Python -m script, the search path starts in the working directory, and not in the bin/ directory. See my answer below for more details.
• Install the package in a virtual environment before using it using setup.py (either run it directly or use pip).
  • This seems redundant if I just test a change that I am not sure, even syntactically correct. Some of the projects I'm working on are not even intended to be installed as packages, but I want to use the same directory structure for everything, and that will mean a setup.py entry so I can test them!
  • Edit: Two interesting options for this are discussed in the answers below: setup.py develop command in logc answer and pip install -e in mine. They avoid the need to “install” for each small edit, but you still need to create setup.py for packages that you never intend to install completely, and doesn't work very well with PyCharm (which has a menu entry to run the develop command, but not an easy way to run scripts that it copies to a virtual environment).
• Move the scripts to the project root directory (i.e. in mylibsrc/ instead of mylibsrc/bin/ ).
  • Ugh! This is a last resort , but unfortunately it seems like the only option available at the moment .

Answer 1

Run modules as scripts

Since I posted this question, I found out that you can run the module as if it were a script using Python -m with the command line (which I thought applies only to packages).

Therefore, I believe that the best solution is the following:

• Instead of writing shell scripts in the bin subdirectory, put the main part of the logic in the modules (as it should be) and put the corresponding modules at the end if __name__ == "__main__": main() , as in the script.
• To run scripts on the command line, call the modules directly as follows: python -m pkg_name.module_name
• If you have setup.py, as Alik said, you can create shell scripts during installation so that your users cannot run them in this funny way.

PyCharm does not support starting modules in this way (see this request ). However, you can simply run modules (as well as scripts in bin), as usual, because PyCharm automatically adds the project root to PYTHONPATH, so import operations are allowed without any extra effort. There are several errors for this:

• The main problem: the working directory will be wrong, so opening data files will not work. Unfortunately, there is no quick fix. the first time you run each script, you must stop it and change its configured working directory (see this link ).
• If the directory of your package is not located directly in the root directory of the project, you need to mark its parent directory as the source directory on the project structure settings page.
• Relative imports do not work, i.e. you can do from pkg_name.other_module import fn , but not from .other_module import fn . In any case, relative imports are generally bad style, but they are useful for unit tests.
• If the module has a cyclic dependency and you start it directly, it will be imported twice (once as pkg_name.module_name and once as __main__ ). But in any case, you should not have circular dependencies.

Command line bonus:

• If you still want to put some scripts in bin/ , you can call them with python -m bin.scriptname (but in Python 2 you need to put __init__.py in the bin directory).
• You can even run a generic package if it has __main__.py , for example: python -m pkg_name

Peak Edit Mode

There is an alternative on the command line that is not so simple, but still worth knowing about:

• Use editable editable mode,
• To use it, create the setup.py file and use the following command to install the package into your virtual environment: pip install -e .
• Notice the endpoint that references the current directory.
• This puts the scripts created from your setup.py file in the bin virtual directory and references the source code of your package, so you can edit and debug it without restarting pip.
• When you are done, you can run pip uninstall pkg_name
• This is similar to the setup.py develop command, but removal seems to work better.

Answer 2

The easiest way is to use setuptools in the setup.py script and use the entry_points keyword, see the Automatic Script Creation documentation.

Details: you create setup.py which looks like this

 from setuptools import setup setup( # other arguments here... entry_points={ 'console_scripts': [ 'foo = my_package.some_module:main_func', 'bar = other_module:some_func', ], 'gui_scripts': [ 'baz = my_package_gui:start_func', ] } ) 

then create other Python packages and modules under the directory where this setup.py exists, for example. following the example above:

 . ├── my_package │  ├── __init__.py │  └── some_module.py ├── my_package_gui │  └── __init__.py ├── other_module.py └── setup.py 

and then run

 $ python setup.py install 

or

 $ python setup.py develop 

In any case, new Python scripts are created for you (executable scripts without the suffix .py ) that point to the entry points that you described in setup.py . They are usually found in the Python interpreter's view of the "directory where the binaries should be executable", which is usually located on your PATH. If you use virtual env, then virtualenv tricks the Python interpreter into thinking that this bin/ directory is where you determined what virtualenv should be. Following the example above in virtualenv, running the previous commands should result in:

 bin ├── bar ├── baz └── foo