How to manage memory with ZODB?

Question

I read several ZODB tutorials, but here is one thing I still don't get: how do you free memory that is already serialized (and committed) in (say) FileStorage?

In particular, I want the following code to not leave all my memory:

for i in xrange(bignumber): iobtree[i]=Bigobject() # Bigobject is about 1Mb if(i%10==0): transaction.commit() # or savepoint(True) transaction.commit() 

How can this be achieved? Is it possible to free links stored in iobtree and replace them with "weak links", which will be available upon request?

Answer 1

Creating savepoints and committing a transaction already clears up a lot of your memory.

• You will need to check what the ZODB cache settings are configured for, and configure them as needed. The cache size parameter indicates the number of cached objects, not bytes, so you will have to adjust this based on the size of your objects.

• You can try and call .cacheMinimize() in the ZODB connection object, this explicitly deactivates any unmodified (or already committed) objects in the cache.

Also, note that even when Python frees objects from memory, the OS does not always recover freed memory until it is needed for something else. OS-OS memory usage does not necessarily reflect the actual memory requirements for the Python process.