Numpy missing attributes depending on array size

Question

Here is a little code that demonstrates the error I am getting.

import numpy as np r=4.0 L=20.0 ratio = 4*np.pi / 3.0 * (r/L)**3 for i in range(5, 16): n = 10**i m = int(ratio * n) print i,n,m ip = np.random.random_integers(100, size=(n,3)) jp = np.random.random_integers(100, size=(m,3)) a = np.expand_dims(ip, -1) == jp.T b = np.where( a.all(axis=1).any(axis=1) )[0] 

I get the following output:

 5 100000 3351 6 1000000 33510 Traceback (most recent call last): File "example.py", line 16, in <module> b = np.where( a.all(axis=1).any(axis=1) )[0] AttributeError: 'bool' object has no attribute 'all' 

Does anyone know what is going on here?

As an alternative, a fairly fast way to index the location of jp elements in ip will also work. I can go with the second solution here

Answer 1

You pass ip against jp , creating extremely large arrays. When i==6 you have an array of 100 GB.

The solution is to loop over the array:

 for i in range(2,6): t=time.time() n = 10**i+1 m = int(ratio * n) print i,n,m ip = np.random.random_integers(10, size=(n,3)) jp = np.random.random_integers(10, size=(m,3)) chunksize=10000 if chunksize>ip.shape[0]: chunksize=ip.shape[0] span=ip.shape[0]/chunksize remainder=(ip.shape[0]-span*chunksize) out=[] start=0 for n in xrange(span): end=start+chunksize a = np.expand_dims(ip[start:end], -1) == jp.T b = np.where( a.all(axis=1).any(axis=1) )[0] out.append(b+start) start+=chunksize if remainder!=0: a = np.expand_dims(ip[-remainder:], -1) == jp.T b = np.where( a.all(axis=1).any(axis=1) )[0] out.append(b+end) end=np.sort(np.concatenate(out)) print time.time()-t,end.shape 

The time i==6 is about 10 seconds, so i==7 will take about 20 minutes.