Creating / testing a Python project with C extensions

Question

I have a project with a python package and a compiled component inside it. Current catalog layout:

<project> foo/ foo/__init__.py foo/... src/ src/c_foo.c tests/ tests/test_foo.py setup.py 

When the project is built, distutils creates the build/lib directory, which I either add to PYTHONPATH or install in a virtual environment. The resulting structure is as follows:

 <project> build/lib build/lib/foo/__init__.py build/lib/foo/c_foo.so 

The problem is that if I start a python interpreter session from the project root, run tests from the project root, etc., it takes the source tree instead of the built-in tree.

I found several existing solutions for this:

• Put the python sources in a separate directory, e.g. lib/foo , modules/foo , etc. The disadvantage of this is the additional directory level for all source files and inconsistency with projects that do not have compiled extensions and therefore have their python packages in the root directory.

• Store packages in the root, which means the inconvenience associated with chdir from the project root (for example, in test / directory), so that the python interpreter does not see the source packages (via build script or manually).

• Keep packages at the root under a different name (for example, foo-module or foo-lib ) with the corresponding line package_dir={'foo':'lib-foo'} in setup.py . This is a variation of pt. 1 without an additional level of directory hierarchy, which, in my opinion, is almost the same.

• Store the packages in the root and use setup.py build_ext --inplace , however this pollutes the source tree.

In any case, overhead information is entered against a simple python project where you can change / run the code directly from the source tree. I would really like to hear all the thoughts about the pros and cons of the above and what specific method you use for your projects.

Answer 1

You might want to try develop target in distribute (previously setuptools) .

Make sure distribute installed, then change your setup.py like this:

 # the setuptools package name is still used from setuptools import setup, Extension ... 

Then enter your virtualenv and run develop :

 % source ~/virt/bin/activate (virt)% cd ~/project (virt)% python setup.py develop 

You should be able to run tests from the project root directory, and at any time when you activate this virtualenv, you can access these project packages and extensions regardless of your path:

 % cd /tmp % source ~/virt/bin/activate (virt)% python -c 'import foo, c_foo; print foo, c_foo' <module 'foo' from '/Users/user/project/foo/__init__.py'> <module 'c_foo' from '/Users/user/project/c_foo.so'>