How to find the total amount of memory used by a python process / object in windows

Question

I have a script that loads a lot of data into memory. I want to know how efficient the data stored in memory is. So, I want to know how much memory python has used before loading data and after loading data. Also I am wondering if this is some way to check the memory usage of a complex object. Let's say I have a nested dictionary with different data types inside. How can I find out how much memory is used by all the data in this dictionary. Thanks, Alex

Answer 1

As far as I know, there is no easy way to find out what memory consumption is for a particular object. This would be a non-trivial thing, because links can be shared between objects.

Here are my two favorite workarounds:

• Use the process manager. Before distributing the program, pause the program. Record the memory used before allocation. Allocate. Record memory after allocation. This is a low tech method, but it works.
• Alternatively, you can use pickle.dumpdata structures to serialize. The resulting brine will be comparable (not identical!) In size with the space needed to store the data structure in memory. For best results, use the binary brine protocol.

Answer 2

guppy , , . , python >= 2.6, , python 2.5. , , :

from guppy import hpy
hp = hpy()
print hp.heap()

:

Partition of a set of 25961 objects. Total size = 1894868 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  11901  46   775408  41    775408  41 str
     1   6040  23   219964  12    995372  53 tuple
     2   1718   7   116824   6   1112196  59 types.CodeType
     3     73   0   113608   6   1225804  65 dict of module
     4    348   1   107232   6   1333036  70 dict (no owner)
     5    196   1   100192   5   1433228  76 dict of type
     6   1643   6    92008   5   1525236  80 function
     7    209   1    90572   5   1615808  85 type
     8    144   1    76800   4   1692608  89 dict of class
     9    984   4    35424   2   1728032  91 __builtin__.wrapper_descriptor

Answer 3

To analyze how much memory an object uses, you can use Pympler :

>>> from pympler import asizeof
>>> obj = dict(nested=dict(trash=[1,2,3]))
>>> asizeof.asizeof(obj)
800
>>> asizeof.asizeof(obj['nested'])
480
>>> asizeof.asizeof(obj['nested']['trash'])
160
>>> asizeof.asizeof(obj['nested']['trash'][0])
24

Answer 4

An alternative is that you can use Windows performance counters through pywin32