Python memory error?

Question

Somehow the memory of my Python program takes up more and more memory when it starts (the VIRT and RES column) of the "top" command continues to grow.

However, I checked my code carefully twice, and I'm sure there are no memory leaks (I didn’t use a dictionary, not global variables. This is just the main method calling the sub method several times),

I used heapy to profile my memory usage with

from guppy import hpy; heap = hpy(); ..... print heap.heap(); 

every time the main method calls the sub method. Surprisingly, it always gives the same result. But memory usage just keeps growing.

I wonder if I used heapy right, or does VIRT and RES in the "top" command really not reflect the memory used by my code?

Or can anyone provide a better way to track memory usage in a Python script?

Thanks a lot!

Answer 1

Two possible cases:

• your function is pure Python, in which case possible causes include

  • you save more and more large objects
  • you have object loops using the __del__ method, which gc will not touch

  I would suggest using the gc module and the gc.garbage and gc.get_objects (see http://docs.python.org/library/gc.html#module-gc ), for a list of existing objects, and then you can use them browse by looking at the __class__ attribute of each object, for example, to get information about the class of the object.

• your function is at least partially written in C / C ++, in which case the problem may be in this code. The above tip is still applicable, but it will not be able to see all the leaks: you will see leaks caused by the absence of PY_DECREF calls, but not low-level C / C ++ distributions without the corresponding release. For this you need valgrind. See this question for more information on this topic.