CPickle.load () in python consumes a lot of memory

Question

I have a large dictionary whose structure looks like this:

dcPaths = {'id_jola_001': CPath instance}

where CPath is a self-defined class:

class CPath(object):
    def __init__(self):
        # some attributes
        self.m_dAvgSpeed = 0.0
        ...
        # a list of CNode instance
        self.m_lsNodes = []

where m_lsNodes is a CNode list:

class CNode(object):
    def __init__(self):
        # some attributes
        self.m_nLoc = 0

        # a list of Apps
        self.m_lsApps = []

Here m_lsApps is a CApp list, which is another self-defined class:

class CApp(object):
    def __init__(self):
        # some attributes
        self.m_nCount= 0
        self.m_nUpPackets = 0

I serialize this dictionary with cPickle:

def serialize2File(strFileName, strOutDir, obj):
    if len(obj) != 0:
        strOutFilePath = "%s%s" % (strOutDir, strFileName)
        with open(strOutFilePath, 'w') as hOutFile:
            cPickle.dump(obj, hOutFile, protocol=0)
        return strOutFilePath
    else:
        print("Nothing to serialize!")

It works great and the serialized file size is about 6.8 GB. However, when I try to deserialize this object:

def deserializeFromFile(strFilePath):
    obj = 0
    with open(strFilePath) as hFile:
        obj = cPickle.load(hFile)
    return obj

I find that it consumes over 90 GB of memory and is time consuming.

• why does this happen?
• Is there any way to optimize this?

By the way, I'm using python 2.7.6

Answer 1

pickle; - -1 ( : , , Python).

cPickle.dump(obj, file, protocol = -1)

: : load .

cPickle.load(obj, file)

Answer 2

python, python ( __dict__).

, python. : object.__getstate__() object.__setstate__(state).

. python.