Odd thread behavior in python

Question

I have a problem when I need to pass the index of an array to a function that I define inline. The function is then passed as a parameter to another function, which will eventually call it as a callback.

The fact is that when the code is called, the index value is incorrect. In the end, I solved this by creating an ugly workaround, but I'm interested in understanding what is going on here. I created a minimal example to demonstrate the problem:

from __future__ import print_function import threading def works_as_expected(): for i in range(10): run_in_thread(lambda: print('the number is: {}'.format(i))) def not_as_expected(): for i in range(10): run_later_in_thread(lambda: print('the number is: {}'.format(i))) def run_in_thread(f): threading.Thread(target=f).start() threads_to_run_later = [] def run_later_in_thread(f): threads_to_run_later.append(threading.Thread(target=f)) print('this works as expected:\n') works_as_expected() print('\nthis does not work as expected:\n') not_as_expected() for t in threads_to_run_later: t.start() 

Here is the result:

 this works as expected: the number is: 0 the number is: 1 the number is: 2 the number is: 3 the number is: 4 the number is: 6 the number is: 7 the number is: 7 the number is: 8 the number is: 9 this does not work as expected: the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 the number is: 9 

Can someone explain what is going on here? I suppose this is due to a covering area or something, but a response with a link explaining this dark (for me) python viewing angle will be valuable to me.

I am running this on python 2.7.11

Answer 1

This is the result of how closures and scope work in python.

What happens is that i bound to the scope of the not_as_expected function. Therefore, despite the fact that you are passing a lambda stream to a stream, the variable that it uses is distributed between each lambda and each stream.

Consider the following example:

 def make_function(): i = 1 def inside_function(): print i i = 2 return inside_function f = make_function() f() 

Which number do you think will be printed? i = 1 before the function was defined or i = 2 after?

The current value of i (i.e. 2 ) will be printed. It does not matter what value i was when the function was executed, it will always use the current value. The same thing happens with your lambda functions.

Even in the expected results, you can see that it does not always work correctly, it skipped 5 and displayed 7 twice. What happens in this case is that each lambda usually works before the loop reaches the next iteration. But in some cases (for example, 5 ), the loop manages to go through two iterations before control is transferred to one of the other threads, and i doubles and the number is skipped. In other cases (for example, 7 ), two threads can be executed while the loop is still at the same iteration, and since i does not change between the two threads, it gets the same value.

If you have done this:

 def function_maker(i): return lambda: print('the number is: {}'.format(i)) def not_as_expected(): for i in range(10): run_later_in_thread(function_maker(i)) 

The variable i bound inside function_maker along with the lambda function. Each lambda function will reference a different variable, and it will work as expected.

Answer 2

A closure in Python captures free variables , not their current values, at the time the closure was created. For instance:

 def make_closures(): L = [] # Captures variable L def push(x): L.append(x) return len(L) # Captures the same variable def pop(): return L.pop() return push, pop pushA, popA = make_closures() pushB, popB = make_closures() pushA(10); pushB(20); pushA(30); pushB(40) print(popA(), popA(), popB(), popB()) 

30, 10, 40, 20 will be displayed: this is because the first pair of closures pushA , popA will refer to one list L , and the second pair of pushB , popB will refer to another independent list.

The important point is that in each pair of push and pop closures refer to the same list, i.e. they capture the variable L , not the value of L at creation time. If L mutated by one closure, then others will see changes.

One common mistake, for example, is to expect that

 L = [] for i in range(10): L.append(lambda : i) for x in L: print(x()) 

displays numbers from 0 to 9 ... all unnamed closures here fix the same variable i that is used for the loop, and they all return the same value when called.

The general Python idiom to solve this problem is

 L.append(lambda i=i: i) 

i.e. using the fact that the default values ​​for parameters are evaluated at the time the function is created. With this approach, each closure returns a different value, because they return their private local variable (a parameter that has a default value).