Fastest way to count array values ​​above threshold in numpy

Question

I have a numpy array containing 10 ^ 8 float and want to calculate how many of them> = a given threshold. Speed ​​is critical since the operation must be performed on a large number of such arrays. Competitors are still

np.sum(myarray >= thresh)

np.size(np.where(np.reshape(myarray,-1) >= thresh))

Responses to Count all the values ​​in a matrix that exceeds the value , suppose np.where () will be faster, but I found inconsistent synchronization results. I mean this for some implementations and Boolean conditions np.size (np.where (cond)) is faster than np.sum (cond), but for some it is slower.

In particular, if most of the records satisfy the condition, then np.sum (cond) is much faster, but if a small part (maybe less than a tenth), then np.size (np.where (cond)) wins,

The question is divided into 2 parts:

• Any other suggestions?
• Does it make sense that the time np.size (np.where (cond)) increases with the number of entries for which cond is true?

Answer 1

Using cython can be a worthy alternative.

import numpy as np
cimport numpy as np
cimport cython
from cython.parallel import prange


DTYPE_f64 = np.float64
ctypedef np.float64_t DTYPE_f64_t


@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
cdef int count_above_cython(DTYPE_f64_t [:] arr_view, DTYPE_f64_t thresh) nogil:

    cdef int length, i, total
    total = 0
    length = arr_view.shape[0]

    for i in prange(length):
        if arr_view[i] >= thresh:
            total += 1

    return total


@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
def count_above(np.ndarray arr, DTYPE_f64_t thresh):

    cdef DTYPE_f64_t [:] arr_view = arr.ravel()
    cdef int total

    with nogil:
       total =  count_above_cython(arr_view, thresh)
    return total

Dates of various suggested methods.

myarr = np.random.random((1000,1000))
thresh = 0.33

In [6]: %timeit count_above(myarr, thresh)
1000 loops, best of 3: 693 µs per loop

In [9]: %timeit np.count_nonzero(myarr >= thresh)
100 loops, best of 3: 4.45 ms per loop

In [11]: %timeit np.sum(myarr >= thresh)
100 loops, best of 3: 4.86 ms per loop

In [12]: %timeit np.size(np.where(np.reshape(myarr,-1) >= thresh))
10 loops, best of 3: 61.6 ms per loop

With a large array:

In [13]: myarr = np.random.random(10**8)

In [14]: %timeit count_above(myarr, thresh)
10 loops, best of 3: 63.4 ms per loop

In [15]: %timeit np.count_nonzero(myarr >= thresh)
1 loops, best of 3: 473 ms per loop

In [16]: %timeit np.sum(myarr >= thresh)
1 loops, best of 3: 511 ms per loop

In [17]: %timeit np.size(np.where(np.reshape(myarr,-1) >= thresh))
1 loops, best of 3: 6.07 s per loop