Python: SWIG vs ctypes

Question

In python, under what circumstances is SWIG better to choose than ctypes for invoking entry points in shared libraries? Suppose you do not have SWIG interface files yet.

What are the performance indicators of the two?

Answer 1

SWIG generates (rather ugly) C or C ++ code. This is a simple use for simple functions (things that can be translated directly) and quite easy to use for more complex functions (such as functions with output parameters, which require an additional translation step for presentation in Python). For more efficient interaction, you often need to write C bits as part of the interface file. For anything other than simple use, you will need to learn about CPython and how it represents objects - not difficult, but something to keep in mind.

ctypes allows you to directly access C functions, structures, and other data and load arbitrary shared libraries. You do not need to write C for this, but you need to understand how C. works. Perhaps you can argue that the flip side is SWIG: it does not generate code, and it does not require a compiler at runtime, but for anything other than simple use , it requires you to understand how things like C data types, memory management, and alignment are. You also need to manually or automatically translate C structures, unions, and arrays into the equivalent ctypes data structure, including the correct memory layout.

It is likely that in pure execution, SWIG is faster than ctypes, because control around the actual work is done in C at compiletime, and not in Python at run time. However, if you do not interact with many different C functions, but each time several times, it is unlikely that the overhead will be really noticeable.

During development, ctypes has a significantly lower startup cost: you do not need to study interface files, you do not need to create .c files and compile them, you do not need to check and disable warnings. You can just jump in and start using one C function with minimal effort, and then expand it to a larger one. And you can test and try directly in the Python interpreter. Wrapping a lot of code is somewhat tedious, although there are attempts to make it easier (e.g. ctypes-configure.)

SWIG, on the other hand, can be used to create wrappers for several languages ​​(ban on language-specific details that need to be filled out, for example, the special C code mentioned above). When packing a large number of codes that SWIG can handle with a little help, code generation can also be much simpler than ctypes equivalents.

Answer 2

I have rich experience using swig. SWIG claims that it is a quick solution for packing things. But in real life ...

---

Minuses:

SWIG is designed as a common language for all and for more than 20 languages. As a rule, this leads to disadvantages:
- configuration is needed (SWIG.i templates), sometimes it’s difficult,
- lack of treatment for some special cases (see python properties below),
- lack of performance for some languages.

Python cons:

1) Code style inconsistency . C ++ and python have very different code styles (this is obvious, of course), the possibilities of swig for creating the target code over pythonish are very limited. As an example, you can create properties from getters and setters. See this q & a

2) Lack of a wide community . Swig has some good documentation. But if someone caught something that is not in the documentation, there is no information at all. No blogs or searches help. Therefore, in such cases, you need to strongly break the code generated by SWIG ... This is terrible, I can say ...

Procs:

• In simple cases, it is very fast, easy and straightforward.

• If you created the swig interface files once, you can wrap this C ++ code with any other 20+ languages ​​(!!!).

• One big problem with SWIG is performance. Since version 2.0.04 SWIG includes the “-builtin” flag, which makes SWIG even faster than other automated packaging methods. At least some tests show this.

---

When to use US SWIG?

So, I concluded for myself two cases when swig is good to use:

2) If you need to wrap C ++ code for several languages . Or, if possible, there may be a time when you need to distribute code for several languages. Using SWIG in this case is reliable.

1) If you need to quickly wrap only a few functions from some C ++ library for end use.

---

Live experience

Update :
A year and a half has passed since we converted our library using SWIG.

First we made a python version. There were a few moments when we ran into problems with SWIG - that’s true. But now we have expanded our library to Java and .NET. So, we have 3 languages ​​with 1 SWIG. And I can say that SWIG rock in terms of time saving.

Update 2 :
This is two years when we use SWIG for this library. SWIG is integrated into our build system. We recently had a big change to the API in C ++. SWIG worked fine. The only thing we needed to do was add a few% renames to the .i files so that our CppCamelStyleFunctions() now looks_more_pythonish in python. At first, I was worried about some problems that might arise, but nothing happened. It was awesome. Just a few changes and everything that is distributed in 3 languages. Now I am sure that this is a good solution for using SWIG in our case.

Update 3 :
This is already 3 years, we use SWIG for our library. Major change : part of python is completely rewritten in pure python. The reason is that python is used for most of the applications in our library. Even if a pure version of python is slower than C ++ packaging, it is more convenient for users to work with pure python without struggling with native libraries.

SWIG is still used for versions of .NET and Java.

The main question is: "Would we use SWIG for python if we started the project from the very beginning?". We will! SWIG allowed us to quickly distribute our product in many languages. It worked for a certain period of time, which gave us the opportunity to better understand the requirements of our users.

Answer 3

CTypes is very cool and much simpler than SWIG, but it has the disadvantage that bad or maliciously written Python code can crash the python process. You should also consider boost python. IMHO this is actually simpler than swig, giving you more control over the final python interface. If you use C ++ anyway, you also do not add any other languages ​​to your mix.

Answer 4

In my experience, ctypes has a big flaw: when something goes wrong (and it will invariably be for any complex interfaces), this is hell for debugging.

The problem is that most of your stack is hidden using ctypes / ffi magic, and there is no easy way to determine how you got to a particular point and why the parameter values ​​are what they are.

Answer 5

You can also use Pyrex , which can act as a glue between high-level Python code and low-level C code. lxml is written, for example, in Pyrex.

Answer 6

I am going to be the opposite and suggest that, if possible, you should write your extension library using the standard Python API . It is really well integrated both in terms of C and Python ... if you have experience with the Perl API, you will find it a very pleasant surprise.

Ctypes is good too, but as others have said, it does not do C ++.

How big is the library you are trying to wrap? How fast does the code base change? Any other service issues? All of this will likely influence the choice of the best way to write Python bindings.

Answer 7

ctypes is fine, but does not handle C ++ classes. I also found that ctypes is about 10% slower than direct C binding, but that will depend a lot on what you call.

If you intend to use ctypes, be sure to check out the Pyglet and Pyopengl projects, which have massive examples of ctype bindings.

Answer 8

Just wanted to add a few more considerations that I haven't mentioned yet. [EDIT: Oh, didn't see Mike Stader answer]

If you want to try using a non-Cpython implementation (e.g. PyPy, IronPython or Jython), then ctypes is the only way. PyPy does not allow writing C-extensions, therefore it excludes pyrex / cython and Boost.python. For the same reason, ctypes is the only mechanism that will work for IronPython and (ultimately, as soon as they work) jython.

As already mentioned, compilation is not required. This means that if a new version of .dll or .so is released, you can simply download it and download this new version. So far, none of the interfaces has changed, this is a reduction in replacement.

Answer 9

Something to keep in mind is that SWIG is only aimed at implementing CPython. Since ctypes is also supported by PyPy and IronPython implementations, it might be worth writing your modules using ctypes for compatibility with the wider Python ecosystem.

Answer 10

I found that SWIG would be a bit bloated in its approach (not just Python in general), and it would be difficult to implement without having to cross the sore point of writing Python code with explicit thinking in order to be SWIG-compatible, rather than writing pure well-written Python code . This, IMHO, is a much simpler process of writing C bindings to C ++ (if using C ++), and then use ctypes to interact with any C layer.

If the library you are interacting with has a C interface as part of the library, another advantage of ctypes is that you do not need to compile a separate python binding library to access third-party libraries. This is especially good when developing a pure-python solution that avoids cross-platform compilation issues (for third-party libraries offered on disparate platforms). Having to embed compiled code in a package that you want to deploy to something like PyPi in a cross-platform way is a pain; one of my most annoying points about Python packages using SWIG or basic C explicit code is their cross-platform inaccessibility. So consider this if you work with cross-platform accessible third-party libraries and develop a python solution around them.

As an example of the real world, consider PyGTK. This (I believe) uses SWIG to generate C code to interact with GTK C calls. I used it for a short time just to find real pain to set up and use, with fancy odd errors if you didn't do anything in the correct order when setting up and just in general. It was such an unpleasant experience, and when I looked at the interace definitions provided by GTK on the Internet, I realized that a simple exercise would be to write an interface translator for the python ctypes interface. A project called PyGGI was created, and one day I was able to rewrite PyGTK as a much more functional and useful product that fully complies with the GT-C object-oriented interfaces. And it does not require compiling C code, which makes it cross-platform. (I was actually after interacting with webkitgtk, which is not so cross-platform). I can also easily deploy PyGGI on any platform that supports GTK.