Pybind11: how to pack C ++ and python code into one package?

Question

I am trying to put together the existing Python code and the new C ++ 11 code using CMake and pybind 11. I think that I am missing something simple to add in the CMake script, but I can not find it anywhere: pybind11 examples have only C ++ and none of Python, other online resources are quite confusing and not updated, so I just can’t figure out how to combine functions in both languages ​​together and make them available through Python import my_package down the line ... as an example, I cloned cmake_example from pybind11 and added multifunction to cmake_example/mult.py

 def mult(a, b): return a * b 

how could I make this visible with add and subtract to pass the test below?

 import cmake_example as m assert m.__version__ == '0.0.1' assert m.add(1, 2) == 3 assert m.subtract(1, 2) == -1 assert m.mult(2, 2) == 4 

this test is currently failing.

Thanks!

Answer 1

The simplest solution has nothing to do with pybind11 as such. What authors usually do when they want to combine pure Python and C / Cython / other native extensions in one package is the following.

You create two modules.

• mymodule is an open interface, pure Python module
• _mymodule is a private implementation, executed module

Then, in mymodule you import the necessary characters from _mymoudle (and, if necessary, back to a clean version of Python).

Here is an example from yarl package:

• quoting.py

   try: from ._quoting import _quote, _unquote quote = _quote unquote = _unquote except ImportError: # pragma: no cover quote = _py_quote unquote = _py_unquote 

• _quoting.pyx

Update

A script follows here. For the sake of reproducibility, I am doing this against the original cmake_example .

 git clone --recursive https://github.com/pybind/cmake_example.git # at the time of writing https://github.com/pybind/cmake_example/commit/8818f493 cd cmake_example 

Now create pure Python modules (inside cmake_example/cmake_example ).

cmake_example/__init__.py

 """Root module of your package""" 

cmake_example/math.py

 def mul(a, b): """Pure Python-only function""" return a * b def add(a, b): """Fallback function""" return a + b try: from ._math import add except ImportError: pass 

Now modify the existing files to turn the cmake_example module into cmake_example._math .

src/main.cpp ( subtract removed for brevity)

 #include <pybind11/pybind11.h> int add(int i, int j) { return i + j; } namespace py = pybind11; PYBIND11_MODULE(_math, m) { m.doc() = R"pbdoc( Pybind11 example plugin ----------------------- .. currentmodule:: _math .. autosummary:: :toctree: _generate add )pbdoc"; m.def("add", &add, R"pbdoc( Add two numbers Some other explanation about the add function. )pbdoc"); #ifdef VERSION_INFO m.attr("__version__") = VERSION_INFO; #else m.attr("__version__") = "dev"; #endif } 

CMakeLists.txt

 cmake_minimum_required(VERSION 2.8.12) project(cmake_example) add_subdirectory(pybind11) pybind11_add_module(_math src/main.cpp) 

setup.py

 # the above stays intact from subprocess import CalledProcessError kwargs = dict( name='cmake_example', version='0.0.1', author='Dean Moldovan', author_email=' [e-mail] ', description='A test project using pybind11 and CMake', long_description='', ext_modules=[CMakeExtension('cmake_example._math')], cmdclass=dict(build_ext=CMakeBuild), zip_safe=False, packages=['cmake_example'] ) # likely there are more exceptions, take a look at yarl example try: setup(**kwargs) except CalledProcessError: print('Failed to build extension!') del kwargs['ext_modules'] setup(**kwargs) 

Now we can build it.

 python setup.py bdist_wheel 

In my case, it produces dist/cmake_example-0.0.1-cp27-cp27mu-linux_x86_64.whl (if C ++ compilation failed cmake_example-0.0.1-py2-none-any.whl ). Here is what this content is ( unzip -l ... ):

 Archive: cmake_example-0.0.1-cp27-cp27mu-linux_x86_64.whl Length Date Time Name --------- ---------- ----- ---- 0 2017-12-05 21:42 cmake_example/__init__.py 81088 2017-12-05 21:43 cmake_example/_math.so 223 2017-12-05 21:46 cmake_example/math.py 10 2017-12-05 21:48 cmake_example-0.0.1.dist-info/DESCRIPTION.rst 343 2017-12-05 21:48 cmake_example-0.0.1.dist-info/metadata.json 14 2017-12-05 21:48 cmake_example-0.0.1.dist-info/top_level.txt 105 2017-12-05 21:48 cmake_example-0.0.1.dist-info/WHEEL 226 2017-12-05 21:48 cmake_example-0.0.1.dist-info/METADATA 766 2017-12-05 21:48 cmake_example-0.0.1.dist-info/RECORD --------- ------- 82775 9 files 

Answer 2

Once you cloned the repo, cd to the top level directory `cmake_example '

Modify. / Src / main.cpp to enable the mult function:

 #include <pybind11/pybind11.h> int add(int i, int j) { return i + j; } int mult(int i, int j) { return i * j; } namespace py = pybind11; PYBIND11_MODULE(cmake_example, m) { m.doc() = R"pbdoc( Pybind11 example plugin ----------------------- .. currentmodule:: cmake_example .. autosummary:: :toctree: _generate add subtract mult )pbdoc"; m.def("add", &add, R"pbdoc( Add two numbers Some other explanation about the add function. )pbdoc"); m.def("mult", &mult, R"pbdoc( Multiply two numbers Some other explanation about the mult function. )pbdoc"); 

(the rest of the file is the same)

Now do this:

 $ cmake -H. -Bbuild $ cmake --build build -- -j3 

The module for import will be created in the directory. / build. Go to it, then in the python shell your example should work.

To import a namespace, you can do something with pkgutil :

create directory structure:

 ./my_mod __init__.py cmake_example.***.so 

and other parallel structure

 ./extensions /my_mod __init__.py cmake_example_py.py 

and put in ./my_mod/__init__.py

 import pkgutil __path__ = pkgutil.extend_path(__path__, __name__) from .cmake_example import add, subtract from .cmake_example_py import mult 

in ./extensions/my_mod/__init__.py

 from cmake_example_py import mult 

Then add both. / my _mod and. / extensions / my_mod to your $ PYTHONPATH, it can work (this is in my example)