Numpy array memory allocation

Question

From what I read about Numpy arrays, they are more memory efficient than standard Python lists. What confuses me is that when you create a numpy array, you need to pass a python list. I assume this python list is deconstructed, but it seems to me that it defeats the goal of creating an efficient data structure with memory if you need to create a large inefficient structure to create an efficient one.

Does numpy.zeros mean this?

Answer 1

There are many ways to create a NumPy array . Passing a Python list to np.array or np.asarray is one such way.

Another way is to use an iterator:

 In [11]: np.fromiter(xrange(10), count=10, dtype='float') Out[11]: array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) 

In this case, there is no large Python temporary list. Therefore, instead of creating a Python list, you can define a generator function that gives the elements in the list. Then, to create an array, you must pass the generator to np.fromiter . Since np.fromiter always creates a 1D array, use reshape for the return value to create arrays with a higher dimension.

There is also np.fromfunction , np.frombuffer , np.fromfile , np.loadtxt , np.genfromtxt , np.fromstring , np.zeros , np.empty and np.ones . All of this provides ways to create NumPy arrays without creating large Python temporary objects.

Answer 2

In the general case, if you pre-select the size, then in the general case this will be more efficient. If you know that you are going to fill the MxN matrix ... first create it and then populate it, as opposed to using add-ons, for example.

While a list needs to be created, most of the efficiency is achieved through the action of this structure. Read / Write / Compute / etc.