GAE has various limitations, one of which is the size of the largest allocated memory block in the amount of 1 MB (now 10 times larger, but this does not change the question). The restriction means that you cannot put more than a certain number of elements in list (), since CPython will try to allocate an adjacent block of memory for element pointers. Having a huge list of () s may be considered bad programming practice, but even if a huge structure is not created in the program itself, CPython supports some behind the scenes.
It seems that CPython supports one global list of objects or something else. That is, an application with many small objects tends to allocate larger and larger individual memory blocks.
The first idea was gc, and disabling it slightly changed the behavior of the application, but some structures are still preserved.
The simplest short application that is having a problem:
a = b = []
number_of_lists = 8000000
for i in xrange(number_of_lists):
b.append([])
b = b[0]
Can someone enlighten me on how to prevent CPython from allocating huge internal structures when there are many objects in the application?
source
share