Dynamically Generated Types Etching

Question

I am trying to get some dynamically created types (i.e. created by calling 3-arg type()) to pickle and expand beautifully. I used this module switch trick to hide details from module users and give clean semantics.

I have already learned a lot:

• The type must be found with getattron the module itself
• The type must correspond to what it finds getattr, that is, if we call pickle.dumps(o), then it must be true, thattype(o) == getattr(module, 'name of type')

Where am I stuck, although it still seems that something strange is happening - it seems that he calls __getstate__something unexpected.

Here is the simplest setup I have, which reproduces the problem by testing it with Python 3.5, but I would like to return to 3.3 if possible:

# module.py
import sys
import functools

def dump(self):
    return b'Some data' # Dummy for testing

def undump(self, data):
    print('Undump: %r' % data) # Do nothing for testing

# Cheaty demo way to make this consistent
@functools.lru_cache(maxsize=None)
def make_type(name):
    return type(name, (), {
        '__getstate__': dump,
        '__setstate__': undump,
    })

class Magic(object):
    def __init__(self, path):
        self.path = path

    def __getattr__(self, name):
        print('Getting thing: %s (from: %s)' % (name, self.path))
        # for simple testing all calls to make_type must end in last x.y.z.last
        if name != 'last':
            if self.path:
                return Magic(self.path + '.' + name)
            else:
                return Magic(name)
        return make_type(self.path + '.' + name)

# Make the switch
sys.modules[__name__] = Magic('')

And then a quick way to implement this:

import module
import pickle

f=module.foo.bar.woof.last()
print(f.__getstate__()) # See, *this* works
print('Pickle starts here')
print(pickle.dumps(f))

Which then gives:

Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
b'Some data'
Pickle starts here
Getting thing: __spec__ (from: )
Getting thing: _initializing (from: __spec__)
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
Getting thing: __getstate__ (from: foo.bar.woof)
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    print(pickle.dumps(f))
TypeError: 'Magic' object is not callable

I did not expect to see something looking up __getstate__at module.foo.bar.woof, but even if we make this search fail, adding:

if name == '__getstate__': raise AttributeError()

in ours __getattr__he still fails:

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    print(pickle.dumps(f))
_pickle.PicklingError: Can't pickle <class 'module.Magic'>: it not the same object as module.Magic

What gives? Am I missing something using __spec__? docs for__spec__ pretty much just emphasizes setting it up appropriately, but doesn't seem to explain very much.

More importantly, the bigger question is, how should I be going to make types that I programmatically generated using the pseudo-module of the __getattr__correct implementation brine?

(, , pickle.dumps -, , pickle.loads undump )

Answer 1

f, pickle f class, module.foo.bar.woof.last.

. :

:

• ...

module.foo.bar.woof.last , module. , , , module.foo.bar.woof, :

    elif parent is not module:
        self.save_reduce(getattr, (parent, lastname))

    else if (parent != module) {
        PickleState *st = _Pickle_GetGlobalState();
        PyObject *reduce_value = Py_BuildValue("(O(OO))",
                                    st->getattr, parent, lastname);
        status = save_reduce(self, reduce_value, NULL);

module.foo.bar.woof . Magic , __getstate__, . Magic , . , .

Answer 2

, , callable - , , , getaround, TYPE. <class 'module.Magic'>: it not the same object as module.Magic, , , self instanciating , , @mock.patch('module.Magic', type(module.Magic)), -.

Main.py

import module
import pickle
import mock


f=module1.foo.bar.woof.last
print(f().__getstate__()) # See, *this* works
print('Pickle starts here')
@mock.patch('module1.Magic', type(module1.Magic))
def pickleit():
    return pickle.dumps(f())
print(pickleit())

class Magic(object):

    def __init__(self, value):
        self.path = value

    __class__: lambda x:x

    def __getstate__(self):
        print ("Shoot me! i'm at " +  self.path )
        return dump(self)

   def __setstate__(self,value):
        print ('something will never occur')
        return undump(self,value)

    def __spec__(self):
        print ("Wrong side of the planet ")

    def _initializing(self):
        print ("Even farther lost ")

     def __getattr__(self, name):
        print('Getting thing: %s (from: %s)' % (name, self.path))
        # for simple testing all calls to make_type must end in last x.y.z.last
        if name != 'last':
            if self.path:
                return Magic(self.path + '.' + name)
            else:
                return Magic(name)
        print('terminal stage' )
        return make_type(self.path + '.' + name)

, , , .