Python freezes for hours after function completion after creating a huge object

Question

I have a function that generates a huge object (about 100-150 GB of memory, on a machine with 500 GB of memory).

The function starts after about 1 hour and writes the file to disk (about 100 MB).

But when the function ends, the program hangs there for several hours without any action (it does not continue the instructions after the place where the function was called).

I suspect that the garbage collector is trying to delete the huge object created in this function, but I can’t see anything (strace doesn’t print anything) and the memory is not decreasing.

Do you have an idea of ​​why this is happening and how to solve it? I am using python 3.5

Answer 1

, , Python, , , ( ). https://mail.python.org/pipermail/python-dev/2008-December/084450.html

, :

• , , , () , .
• , , , . , , .
  • , (1-2 ), , , .

, . ( ). https://mail.python.org/pipermail/python-dev/2008-December/084560.html

, , , , !