What is the fastest way to extract row and column data from Numpy ndarray?

Question

I have a large square matrix (approximately 14,000 x 14,000) represented as a Numpy ndarray . I want to extract a large number of rows and columns - indexes that I know about in advance, although in fact they will be all rows and columns that are not all zero to get a new square matrix (approximately 10,000 x 10000).

The fastest way I've found is:

 > timeit A[np.ix_(indices, indices)] 1 loops, best of 3: 6.19 s per loop 

However, this is much slower than the time spent on matrix multiplication:

 > timeit np.multiply(A, A) 1 loops, best of 3: 982 ms per loop 

This seems strange, because to extract a row / column and multiply the matrix, you need to select a new array (which will be even larger for the result of multiplying the matrix than to extract), but to multiply the matrix, you also need to perform additional calculations.

Thus, the question arises: is there a more efficient way to perform extraction, in particular, which is at least as fast as matrix multiplication?

Answer 1

If I try to reproduce your problem, I do not see such a drastic effect. I notice that depending on how many indexes you choose, indexing can be even faster than multiplication.

 >>> import numpy as np >>> np.__version__ Out[1]: '1.9.0' >>> N = 14000 >>> A = np.random.random(size=[N, N]) >>> indices = np.sort(np.random.choice(np.arange(N), 0.9*N, replace=False)) >>> timeit A[np.ix_(indices, indices)] 1 loops, best of 3: 1.02 s per loop >>> timeit A.take(indices, axis=0).take(indices, axis=1) 1 loops, best of 3: 1.37 s per loop >>> timeit np.multiply(A,A) 1 loops, best of 3: 748 ms per loop >>> indices = np.sort(np.random.choice(np.arange(N), 0.7*N, replace=False)) >>> timeit A[np.ix_(indices, indices)] 1 loops, best of 3: 633 ms per loop >>> timeit A.take(indices, axis=0).take(indices, axis=1) 1 loops, best of 3: 946 ms per loop >>> timeit np.multiply(A,A) 1 loops, best of 3: 728 ms per loop