How do landscaping work?

Question

How are greenlets implemented? Python uses the C stack for the interpreter, and heap allocates the frames of the Python stack, but apart from that, how does it allocate / swap stacks, how does it connect to the interpreter and function mechanisms, and how does it interact with C extensions? (Any quirks)?

There are some comments at the top of greenlet.c in the source, but they are a bit opaque. FWIW I proceed from the perspective of someone who is not familiar with the internal components of CPython, but is familiar with low-level system programming, C, threads, events, coroutines / shared threads, kernel programming, etc.

(Some data points: they do not use ucontext.h , and they do 2x memcpy, highlight, and freely on each context switch .)

Answer 1

When the python program starts, you have essentially two pieces of code running under the hood.

First, the C code of the CPython interpreter is executed and the standard C-stitch is used to save its internal stack frames. Secondly, the actual python interprets the bytecode, which does not use the C-stack, but uses a bunch to save its stack frames. Gellet is just standard python code and therefore behaves the same.

Now in a typical microflow application, you will have thousands, if not millions, of microtypes (green) that switch all over the place. Each switch is essentially equivalent to calling a deferred return function (so to speak) and thus will use the stack bit. The problem is that the interpreter's C stack will sooner or later hit the stack overflow. This is exactly what the green extension is aimed at; it is designed to move pieces of the stack back and forth to / from the heap to avoid this problem.

As you know, there are three fundamental events with greens, caviar, switch and reset, so let's look at them:

A) spawn

Recently appeared greens are associated with their own base address on the stack (where we are currently). In addition, nothing special happens. The python code of the newly created green dot uses the heap in normal mode, and the interpreter continues to use the C stack as usual.

B) switch

When the green panel switches from the green patch panel, the corresponding part of the C-stack (starting from the base address of the patch panel) is copied to the heap. The copied C-stack area is freed, and the previously saved stack file switches from the heap to the newly freed C-stack area. The python dial-up code continues to use the heap in the usual way. Of course, the extension code keeps track of all of this (which section of the heap matches any rose, etc.).

C) return

The stack is untouched, and the heap area of ​​the returning shard is freed by the python garbage collector.

In essence, this is a lot more details and explanations can be found at ( http://www.stackless.com/pipermail/stackless-dev/2004-March/000022.html ) or simply by reading the code as indicated in Alex's answer.

Answer 2

If you get and explore the sources of greenlet , you will see at the top of greenlet.c long comment that starts on line 16 with the following summary ...:

A PyGreenlet is a set of stacks of C addresses that must be saved and restored in such a way that the full range of the stack contains reliable data when we switch to it.

and continues line 82, which indicates exactly what you are asking. Have you studied these lines (and the following 1000+ of their implementations ;-) ...? I do not see the possibility of further squeezing these 66 lines down, still making sense, and no additional value for copying and pasting them here.

In principle, you will see that there is no real “link” that can be talked about (for example, level C stacks switch back and forth “under the nose of the interpreter”), with the exception of subtle interactions with the state of the stream in a multi-threshold code, and saving and greenlet recovery state from / to the stack based on the call memcpy plus some calls Python memory controller for distribution / redistribution and free space emitted from a stack, or vice versa, three functions in the line 227-295 treated work grunt, and they were wrapped in a pair of macros C at 298-310 ", to simplify maintenance," as they say in the comments.

The interface through which other C extensions can interact with the greenlet extension is implemented on lines 956-1045 and viewed through the "CObject API" (via greenlet.h , of course), documented here .