Cython iterates over a list of numpy arrays without gil

Question

I would like to iterate over a list of numpy arrays with different dimensions and pass them to a cython function that does not require GIL:

# a has T1 rows and M columns
a = np.array([[0.0, 0.1, 0.3, 0.7],
              [0.1, 0.2, 0.1, 0.6],
              [0.1, 0.2, 0.1, 0.6]])

# b has T2 rows and M columns
b = np.array([[1.0, 0.0, 0.0, 0.0],
              [0.1, 0.2, 0.1, 0.6]])

# c has T3 rows and M columns
c = np.array([[0.1, 0.0, 0.3, 0.6],
              [0.5, 0.2, 0.3, 0.0],
              [0.0, 1.0, 0.0, 0.0],
              [0.0, 0.0, 0.1, 0.0]])

array_list = [a, b, c]
N = len(array_list)

# this function works
@cython.boundscheck(False)
@cython.wraparound(False)
cdef void function_not_requiring_the_gil(double[:, ::1] arr) nogil:
    cdef int T = arr.shape[0]
    cdef int M = arr.shape[1]
    cdef int i, t

    for t in range(T):
        for i in range(M):
            # do some arbitrary thing to the array in-place
            arr[t, i] += 0.001

# issue is with this function
def function_with_loop_over_arrays(array_list, int N):
    cdef int i

    with nogil:
        for i in range(N):
            function_not_requiring_the_gil(array_list[i])

When I compile Cython code, I get the following error:

Error compiling Cython file:
------------------------------------------------------------
...
def function_with_loop_over_arrays(array_list, int N):
    cdef int i

    with nogil:
        for i in range(N):
            function_not_requiring_the_gil(array_list[i])                                                    ^
------------------------------------------------------------

my_file.pyx:312:53: Indexing Python object not allowed without gil

Is there any other type of data structure that I can use instead of a Python list to store these numpy arrays so that I can iterate over them without gil? I am open to suggestions related to malloc C / Cython memoryviews / other types of pointers that I don't know about.

Note that each numpy array has a different number of lines, but the length of the list of arrays is known.

Answer 1

(3,) function_with_loop_over_arrays: (array_starts), a, b c, (arrays_rows), T1, T2 T3.

function_not_requiring_the_gil , , function_with_loop_over_arrays :

for i in range(N):  # N is 3 and should be passed to function_with_loop_over_arrays
    function_not_requiring_the_gil(array_starts[i], array_rows[i])  

Answer 2

, Python list gil, , . nogil,

def function_with_loop_over_arrays(array_list, int N):
    cdef int i
    cdef double[:, ::1] tmp_mview

    for i in range(N):
        tmp_mview = array_list[i]
        with nogil:        
            function_not_requiring_the_gil(tmp_mview)

( with gil:.)

gil, , function_not_requiring_the_gil , .

Answer 3

nogilfunctions are often easy to control from numba :

import numba
@numba.jit(nogil=True) 
def function_not_requiring_the_gil(arr):
    T,N = arr.shape
    for t in range(T):
        for i in range(N):
        # do some arbitrary thing to the array in-place
        arr[t, i] += 0.001

def function_with_loop_over_arrays(array_list)
        for a in array_list:
            function_not_requiring_the_gil(a)

will give you the same (effective) result.