Abnormal concatenation is slow: any alternative approach?

Question

I run the following code:

for i in range(1000)
    My_Array=numpy.concatenate((My_Array,New_Rows[i]), axis=0)

The above code is slow. Is there a faster approach?

Answer 1

This is basically what happens in all array based algorithms.

Each time you resize an array, it must be resized and every element must be copied. This is happening here. (some implementations reserve several empty slots; for example, doubles the internal memory space at each increase).

• If you received data while creating np.array, just add them all at once (memory will be allocated only once!)
• , - ( O (1) appending-). np.array ( ).

, numpy, .

: , , . , Python ( ). , list.append() python (: O (1)), numpy- ! :

?

Pythons - , Lisp. .

a [i] , .

, . ; , , .

( )

Answer 2

, New_Rows[i] , . ( 1d ), ( 1d), . Concatenate , 2 .

 np.concatenate(New_Rows, axis=0)

, , ( )

 np.concatenate([row for row in New_Rows])

.

 np.concatenate([New_Rows[i] for i in range(1000)])

New_Rows , 2d, New_Rows , np.array :

 np.array(New_Rows)
 np.array([i for i in New_Rows])
 np.array([New_Rows[i] for i in range(1000)])

np.array .

np.concatenate 2d, 2d. vstack stack . stack - , concatenate.

, np.array ( ) . appending ; , .

Answer 3

, , ? ,

import numpy as np 
arr = np.zeros((len(l),)+l[0].shape) 
for i, v in enumerate(l):
   arr[i] = v

,

Answer 4

Suppose you have a large list of two-dimensional arrays with the same number of columns and different number of rows, for example:

x = [numpy_array1 (r_1, c), ......, numpy_arrayN (r_n, c)]

combine like this:

while len(x) != 1:
    if len(x) == 2:
        x = np.concatenate((x[0], x[1]))
        break
    for i in range(0, len(x), 2):
        if (i+1) == len(x):
            x[0] = np.concatenate((x[0], x[i]))
        else:
            x[i] = np.concatenate((x[i], x[i+1]))

    x = x[::2]