Effectively creating numpy arrays from list comprehension and overall

Question

In my current job, I use Numpy and list methods many times, and in the interest of best performance, I have the following questions:

What actually happens behind the scenes if I create a Numpy array as follows?

a = numpy.array( [1,2,3,4] ) 

I assume that python first creates a regular list containing the values, then uses the size of the list to highlight the numpy array, and then copies the values ​​to this new array. Is this correct, or is the interpreter smart enough to understand that the list is only intermediate and instead copy the values ​​directly?

Similarly, if I want to create a numpy array from a list comprehension using numpy.fromiter ():

 a = numpy.fromiter( [ x for x in xrange(0,4) ], int ) 

Will this result lead to an intermediate list of generated values ​​before being sent to fromiter ()?

Regards Niels

Answer 1

I believe the answer you are looking for is using generator expressions with numpy.fromiter .

 numpy.fromiter((<some_func>(x) for x in <something>),<dtype>,<size of something>) 

Generator expressions are lazy - they evaluate the expression when you iterate over them.

Using lists in a list makes a list and then passes it to numpy, while the generator expressions will give one at a time.

Python evaluates things inside → out, like most languages ​​(if not all), so using [<something> for <something_else> in <something_different>] will make a list and then iterate over it.

Answer 2

You can create your own list and experiment with it to shed light on the situation ...

 >>> class my_list(list): ... def __init__(self, arg): ... print 'spam' ... super(my_list, self).__init__(arg) ... def __len__(self): ... print 'eggs' ... return super(my_list, self).__len__() ... >>> x = my_list([0,1,2,3]) spam >>> len(x) eggs 4 >>> import numpy as np >>> np.array(x) eggs eggs eggs eggs array([0, 1, 2, 3]) >>> np.fromiter(x, int) array([0, 1, 2, 3]) >>> np.array(my_list([0,1,2,3])) spam eggs eggs eggs eggs array([0, 1, 2, 3])