How do Python generators know who is calling?

Question

This question makes me pull my hair out.

if:

def mygen(): for i in range(100): yield i 

and call it from thousands of threads, how does the generator know what to send next for each thread? Every time I call it, does the generator save a table with a counter and a link to the caller, or something like that?

This is strange.

Please clarify my mind on this.

Answer 1

mygen does not need to remember anything. Each call to mygen() returns an independent iterable. On the other hand, these iterations have a state: every time next() is called one, it goes to the right place in the generator code - when yield is encountered, control is transferred to the caller. The actual implementation is rather messy, but in principle you can imagine that such an iterator stores local variables, bytecode and the current position in bytecode (aka instruction pointer). There is nothing special here.

Answer 2

A function like this, when called, will return a generator object. If you have separate threads calling next() on the same generator object, they will interfere with each other. That is, 5 threads calling next() 10 times each will receive 50 different outputs.

If two threads create a generator by calling mygen() inside the stream, they will have separate generator objects.

A generator is an object, and its state will be stored in memory, so two threads, each of which creates mygen() , will refer to separate objects. This is no different from two threads creating an object from a class , each of them will have a different object, although the class is the same.

if you use this in the background of C, this is not the same as a function with static variables. State is maintained in the object, not statically in the variables contained in the function.

Answer 3

It might be more clear if you look at it that way. Instead:

 for i in mygen(): . . . 

using:

 gen_obj = mygen() for i in gen_obj: . . . 

then you can see that mygen () is called only once, and it creates a new object, and it is this object that gets iterated. You could create two sequences in the same thread if you wanted:

 gen1 = mygen() gen2 = mygen() print(gen1.__next__(), gen2.__next__(), gen1.__next__(), gen2.__next__()) 

This will print 0, 0, 1, 1.

You can access the same iterator from two threads, if you want, just save the generator object in the global:

 global_gen = mygen() 

Theme 1:

 for i in global_gen: . . . 

Theme 2:

 for i in global_gen: . . . 

This is likely to cause all kinds of chaos. :-)