Python C library SWIG interface (creating an "iterable" Python data type from a C sequence structure)

Question

I wrote a Python extension for the C library. I have a data structure that looks like this:

typedef struct _mystruct{ double * clientdata; size_t len; } MyStruct; 

The purpose of this data type is mapped directly to the list data type in Python. Therefore, I want to create a list-like behavior for the exported structure, so the code written using my C extension is larger than Pythonic.

In particular, this is what I want to do (from python code) Note: py_ctsruct is the ctsruct data type that is accessed in python.

My requirements can be summarized as:

• list (py_ctsruct) returns a python list with all the contents copied from c struct
• py_cstruct [i] returns an ith element (preferably generates an IndexError at an invalid index)
• for elem in py_ctsruct: listing option

According to PEP234 , an Object can be renamed with "for" if it implements _iter_ () or _getitem _ () . Using this logic, I think that by adding the following attributes (via rename ) to my SWIG interface file, I will have the desired behavior (except for the above number 1, which I still don't know how to achieve):

 __len__ __getitem__ __setitem__ 

Now I can index the C object in python. I have not yet implemented Python exception throwing, however, if the array bounds are exceeded, a magic number is returned (error code).

Interestingly, when I try to iterate through the structure using the "for x in" syntax, for example:

 for i in py_cstruct: print i 

Python enters an infinite loop that simply prints the magic number (error) mentioned above on the console. which tells me something is wrong with indexing.

Last but not least, how can I implement requirement 1? this includes (as I understand it):

• handling 'call to list () function from python
• Python (list) data type return from C code

[[Update]]

I would be interested to see a small piece of code about which (if any) declarations I need to put in the interface file so that I can iterate over c struct elements from Python.

Answer 1

The easiest solution is to implement __getitem__ and throw an IndexError exception for the invalid index.

I have compiled an example of this using %extend and %exception in SWIG to implement __getitem__ and raise an exception accordingly:

 %module test %include "exception.i" %{ #include <assert.h> #include "test.h" static int myErr = 0; // flag to save error state %} %exception MyStruct::__getitem__ { assert(!myErr); $action if (myErr) { myErr = 0; // clear flag for next time // You could also check the value in $result, but it a PyObject here SWIG_exception(SWIG_IndexError, "Index out of bounds"); } } %include "test.h" %extend MyStruct { double __getitem__(size_t i) { if (i >= $self->len) { myErr = 1; return 0; } return $self->clientdata[i]; } } 

I tested it by adding to test.h:

 static MyStruct *test() { static MyStruct inst = {0,0}; if (!inst.clientdata) { inst.len = 10; inst.clientdata = malloc(sizeof(double)*inst.len); for (size_t i = 0; i < inst.len; ++i) { inst.clientdata[i] = i; } } return &inst; } 

And run the following Python:

 import test for i in test.test(): print i 

What prints:

 python run.py 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 

and then ends.

---

An alternative approach is also possible using a type map to map MyStruct to PyList :

 %module test %{ #include "test.h" %} %typemap(out) (MyStruct *) { PyObject *list = PyList_New($1->len); for (size_t i = 0; i < $1->len; ++i) { PyList_SetItem(list, i, PyFloat_FromDouble($1->clientdata[i])); } $result = list; } %include "test.h" 

This will create a PyList with a return value from any function that returns MyStruct * . I tested this %typemap(out) with the same function as the previous method.

You can also write the corresponding %typemap(in) and %typemap(freearg) for the reverse, something like this unverified code:

 %typemap(in) (MyStruct *) { if (!PyList_Check($input)) { SWIG_exception(SWIG_TypeError, "Expecting a PyList"); return NULL; } MyStruct *tmp = malloc(sizeof(MyStruct)); tmp->len = PyList_Size($input); tmp->clientdata = malloc(sizeof(double) * tmp->len); for (size_t i = 0; i < tmp->len; ++i) { tmp->clientdata[i] = PyFloat_AsDouble(PyList_GetItem($input, i)); if (PyErr_Occured()) { free(tmp->clientdata); free(tmp); SWIG_exception(SWIG_TypeError, "Expecting a double"); return NULL; } } $1 = tmp; } %typemap(freearg) (MyStruct *) { free($1->clientdata); free($1); } 

---

Using an iterator will make more sense for containers like linked lists, but for completeness you can do this for MyStruct with __iter__ . The key bit is that you get SWIG for another type of transfer for you, which provides __iter__() and next() necessary, in this case MyStructIter , which is determined and MyStructIter at the same time using %inline , since it does not is part of the regular C API:

 %module test %include "exception.i" %{ #include <assert.h> #include "test.h" static int myErr = 0; %} %exception MyStructIter::next { assert(!myErr); $action if (myErr) { myErr = 0; // clear flag for next time PyErr_SetString(PyExc_StopIteration, "End of iterator"); return NULL; } } %inline %{ struct MyStructIter { double *ptr; size_t len; }; %} %include "test.h" %extend MyStructIter { struct MyStructIter *__iter__() { return $self; } double next() { if ($self->len--) { return *$self->ptr++; } myErr = 1; return 0; } } %extend MyStruct { struct MyStructIter __iter__() { struct MyStructIter ret = { $self->clientdata, $self->len }; return ret; } } 

The requirements for iterating over containers are such that the container needs to implement __iter__() and return a new iterator, but in addition to next() , which returns the next element and increments the iterator, the iterator itself must also provide the __iter__() method. This means that either the container or the iterator can be used the same way.

MyStructIter should track the current state of the iteration - where are we and how much is left. In this example, I did this by pointing to the next element and the counter we use to indicate when we got to the end. You could also keep track of the state by storing a pointer to MyStruct , which uses an iterator, and a counter for the position inside that, something like:

 %inline %{ struct MyStructIter { MyStruct *list; size_t pos; }; %} %include "test.h" %extend MyStructIter { struct MyStructIter *__iter__() { return $self; } double next() { if ($self->pos < $self->list->len) { return $self->list->clientdata[$self->pos++]; } myErr = 1; return 0; } } %extend MyStruct { struct MyStructIter __iter__() { struct MyStructIter ret = { $self, 0 }; return ret; } } 

(In this case, we could just use the container itself as an iterator as an iterator by providing __iter__() , which returned a copy of the container and next() similar to the first type. T do this in my original answer, because I thought it would be less clear than two different types - container and iterator for this container)

Answer 2

• Take a look using the cmig command. http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn25 Member map memberin can do what you want. http://www.swig.org/Doc2.0/SWIGDocumentation.html#Typemaps_nn35 I have a sample map that I found in the Python section that allows me to wrap char ** data in C ++ as a list of Python strings. I would suggest that there will be similar functionality.
• Alternatively, you can define% pythoncode in your interface inside the structure inside the "i" swig file. This will allow you to add python methods to the object that is created for the structure. There is another% addmethod command (I think) that allows you to add methods to a structure or class. Then you can create methods for indexing objects in C ++ or C, if you want. There are many ways to solve this problem.

For the interface I'm working on, I used a class object that has some methods for accessing data in my code. These methods are written in C ++. Then I used the% pythoncode directive inside the class inside the “i” file and created the “ getitem ” and “ setitem ” methods in Python code, which uses C ++ methods to expand it to look like accessing a dictionary style.

Answer 3

I ran into the same issue with Python 2.6 and solved it thanks to @aphex's answer. But I wanted to avoid some kind of magic value or additional logical state to complete the state of the end of the list. Of course, my iterator has atEnd () methods that tell me that I'm minus the end of the list.

Thus, it is pretty easy with handling SWIG exceptions. I just had to add the following magic:

 %ignore MyStructIter::atEnd(); %except MyStructIter::next { if( $self->list->atEnd() ) { PyErr_SetString(PyExc_StopIteration,"End of list"); SWIG_fail; } $action } 

The bottom line is that this sniper will miss the following () calls after you finish the end of the list.

If you stick with your idioms, it should look like this:

 %except MyStructIter::next { if( $self->pos >= $self->list->len ) { PyErr_SetString(PyExc_StopIteration,"End of list"); SWIG_fail; } $action } 

NOTE FOR PYTHON 3.x:

You should call the next () function the magic prefix "__" and the postfix name. One option is to simply add:

 %rename(__next__) MyStructIter::next; 

Answer 4

You say that you have yet to implement Python throwing - this is the problem. From PEP 234:

A new exception is defined, StopIteration, which can be used to signal the end of an iteration.

You should set this exception at the end of your iteration. Since your code does not do this, you are faced with a situation that you described:

• The interpreter iternext over your custom iternext list function
• Your function goes to the end of the array, and instead of setting the exception StopIteration simply returns your magic number.
• The interpreter, not noticing the reasons for ending the iteration, simply continues to print the value returned by iternext ... with your magic number. For the translator, this is just another member of the list.

Fortunately, this is a fairly simple fix, but it may not seem so simple, because C has no exceptions. The Python C API simply uses the global error indicator, which you set when an exception occurs, and then the API standards dictate that you return NULL all the way to the stack to the interpreter, which then looks at the output of PyErr_Occurred() to check if an error, and if so, prints the appropriate exception and trace.

So, in your function, when you reach the end of the array, you just need to:

 PyErr_SetString(PyExc_StopIteration,"End of list"); return NULL; 

Here is another great answer for further reading on this issue: How to create a generator / iterator with the Python C API?