Are syntax sugar enumerations for `list (generator expression)` in Python 3?

Question

In Python 3, just syntactic sugar is understood to express the generator entered in the list function?

eg. this is the following code:

 squares = [x**2 for x in range(1000)] 

actually converted in the background to the following?

 squares = list(x**2 for x in range(1000)) 

I know that the output is identical, and Python 3 corrects unexpected side effects for the surrounding namespaces, which are listed in the understanding, but in terms of what the CPython interpreter does under the hood, is the first converted to the last, or is there a difference in that how is the code executed?

Background

I found this equivalence requirement in the comments section for this question , and a quick Google search showed that the same requirement was made here .

There was also some mention of this in What New in Python 3.0 docs , but the wording is somewhat vague:

Also note that the list views have different semantics: they are closer to syntactic sugar for expressing the generator inside the list () constructor, and, in particular, loop control variables no longer leak into the surrounding volume.

Answer 1

Both work differently, in the version for understanding the list a special bytecode LIST_APPEND , which calls PyList_Append directly for us. Therefore, it avoids searching for the list.append attribute and calling the function at the Python level.

 >>> def func_lc(): [x**2 for x in y] ... >>> dis.dis(func_lc) 2 0 LOAD_CONST 1 (<code object <listcomp> at 0x10d3c6780, file "<ipython-input-42-ead395105775>", line 2>) 3 LOAD_CONST 2 ('func_lc.<locals>.<listcomp>') 6 MAKE_FUNCTION 0 9 LOAD_GLOBAL 0 (y) 12 GET_ITER 13 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 16 POP_TOP 17 LOAD_CONST 0 (None) 20 RETURN_VALUE >>> lc_object = list(dis.get_instructions(func_lc))[0].argval >>> lc_object <code object <listcomp> at 0x10d3c6780, file "<ipython-input-42-ead395105775>", line 2> >>> dis.dis(lc_object) 2 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 16 (to 25) 9 STORE_FAST 1 (x) 12 LOAD_FAST 1 (x) 15 LOAD_CONST 0 (2) 18 BINARY_POWER 19 LIST_APPEND 2 22 JUMP_ABSOLUTE 6 >> 25 RETURN_VALUE 

On the other hand, the version of list() simply passes the generator object to the __init__ list, which then calls its extend inside. Since the object is not a list or tuple of CPython, it first gets its iterator , and then simply adds the items to the list until the Iterator is exhausted :

 >>> def func_ge(): list(x**2 for x in y) ... >>> dis.dis(func_ge) 2 0 LOAD_GLOBAL 0 (list) 3 LOAD_CONST 1 (<code object <genexpr> at 0x10cde6ae0, file "<ipython-input-41-f9a53483f10a>", line 2>) 6 LOAD_CONST 2 ('func_ge.<locals>.<genexpr>') 9 MAKE_FUNCTION 0 12 LOAD_GLOBAL 1 (y) 15 GET_ITER 16 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 19 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 22 POP_TOP 23 LOAD_CONST 0 (None) 26 RETURN_VALUE >>> ge_object = list(dis.get_instructions(func_ge))[1].argval >>> ge_object <code object <genexpr> at 0x10cde6ae0, file "<ipython-input-41-f9a53483f10a>", line 2> >>> dis.dis(ge_object) 2 0 LOAD_FAST 0 (.0) >> 3 FOR_ITER 15 (to 21) 6 STORE_FAST 1 (x) 9 LOAD_FAST 1 (x) 12 LOAD_CONST 0 (2) 15 BINARY_POWER 16 YIELD_VALUE 17 POP_TOP 18 JUMP_ABSOLUTE 3 >> 21 LOAD_CONST 1 (None) 24 RETURN_VALUE >>> 

Time comparison:

 >>> %timeit [x**2 for x in range(10**6)] 1 loops, best of 3: 453 ms per loop >>> %timeit list(x**2 for x in range(10**6)) 1 loops, best of 3: 478 ms per loop >>> %%timeit out = [] for x in range(10**6): out.append(x**2) ... 1 loops, best of 3: 510 ms per loop 

Normal loops are a bit slow due to slow attribute lookups. Cache again and again.

 >>> %%timeit out = [];append=out.append for x in range(10**6): append(x**2) ... 1 loops, best of 3: 467 ms per loop 

---

Besides the fact that list comprehension does not supplant variables, another difference is that something like this is no longer valid:

 >>> [x**2 for x in 1, 2, 3] # Python 2 [1, 4, 9] >>> [x**2 for x in 1, 2, 3] # Python 3 File "<ipython-input-69-bea9540dd1d6>", line 1 [x**2 for x in 1, 2, 3] ^ SyntaxError: invalid syntax >>> [x**2 for x in (1, 2, 3)] # Add parenthesis [1, 4, 9] >>> for x in 1, 2, 3: # Python 3: For normal loops it still works print(x**2) ... 1 4 9 

Answer 2

Both forms create and invoke an anonymous function. However, the form list(...) creates a generator function and passes the returned generator iterator to list , and with the form [...] anonymous function builds the list directly using the LIST_APPEND codes.

The following code gets the result of decompiling anonymous functions to understand the example and its corresponding genexp-pass-to- list :

 import dis def f(): [x for x in []] def g(): list(x for x in []) dis.dis(f.__code__.co_consts[1]) dis.dis(g.__code__.co_consts[1]) 

Way out for understanding

  4 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 12 (to 21) 9 STORE_FAST 1 (x) 12 LOAD_FAST 1 (x) 15 LIST_APPEND 2 18 JUMP_ABSOLUTE 6 >> 21 RETURN_VALUE 

The output for the xp gene is

  7 0 LOAD_FAST 0 (.0) >> 3 FOR_ITER 11 (to 17) 6 STORE_FAST 1 (x) 9 LOAD_FAST 1 (x) 12 YIELD_VALUE 13 POP_TOP 14 JUMP_ABSOLUTE 3 >> 17 LOAD_CONST 0 (None) 20 RETURN_VALUE