Numpy.array (list) slow

Question

So, I have a list with 5,000,000 integers. And I want to span the list up to a numpy array. I tried the following code:

numpy.array( list )

But it is very slow.

I compared this operation 100 times and iterated over the list 100 times. There is not much difference.

Any good idea how to make it faster?

Answer 1

If you have cython , you can create a specific function. But just a warning: it will fail if there are invalid elements in your list (not integers or integers that are too large).

IPython (%load_ext cython %%cython), , , - , Cython- ( Cythons) " ).

%load_ext cython

%%cython

cimport cython
import numpy as np

@cython.boundscheck(False)
cpdef to_array(list inp):
    cdef long[:] arr = np.zeros(len(inp), dtype=long)
    cdef Py_ssize_t idx
    for idx in range(len(inp)):
        arr[idx] = inp[idx]
    return np.asarray(arr)

:

import numpy as np

def other(your_list):  # the approach from @Damian Lattenero in the other answer
    ret = np.zeros(shape=(len(your_list)), dtype=int)
    np.copyto(ret, your_list)
    return ret

inp = list(range(1000000))
%timeit np.array(inp)
# 315 ms ± 5.42 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit np.array(inp, dtype=int)
# 311 ms ± 2.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit other(inp)
# 316 ms ± 3.97 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit to_array(inp)
# 23.4 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

10 .

Answer 2

, , :

import numpy as np
import time
start_time = time.time()

number = 1
elements = 10000000

your_list = [number] * elements

ret = np.zeros(shape=(len(your_list)))
np.copyto(ret, your_list)

print("--- %s seconds ---" % (time.time() - start_time))
--- 0.7615997791290283 seconds ---

Answer 3

; numpy:

In [619]: arr = np.random.randint(0,256, 5000000)
In [620]: alist = arr.tolist()
In [621]: timeit alist = arr.tolist()     # just for reference
10 loops, best of 3: 108 ms per loop

( )

In [622]: timeit [i for i in alist]
10 loops, best of 3: 193 ms per loop

dtype

In [623]: arr8 = np.array(alist, 'uint8')
In [624]: timeit arr8 = np.array(alist, 'uint8')
1 loop, best of 3: 508 ms per loop

2x fromiter; , . np.array , . ..

In [625]: timeit arr81 = np.fromiter(alist, 'uint8')
1 loop, best of 3: 249 ms per loop

, :

In [628]: timeit arr8.sum()
100 loops, best of 3: 6.93 ms per loop
In [629]: timeit sum(alist)
10 loops, best of 3: 74.4 ms per loop
In [630]: timeit 2*arr8
100 loops, best of 3: 6.89 ms per loop
In [631]: timeit [2*i for i in alist]
1 loop, best of 3: 465 ms per loop

It is well known that working with arrays is faster than with lists, but there is significant overhead to launch.