Brine debugging

Question

I am trying to sort a fairly involved hierarchy of objects and get an exception:

pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed 

Are there any reasonable methods that can be used to verify the disassembly of the hierarchy of objects? My goal would be to find the location of the disturbing function

Answer 1

To do this, I would use a dill that can serialize almost anything in python. Dill also has some good tools to help you understand what causes your etching to fail when your code crashes.

 >>> import dill >>> dill.loads(dill.dumps(your_bad_object)) >>> ... >>> # if you get a pickling error, use dill tools to figure out a workaround >>> dill.detect.badobjects(your_bad_object, depth=0) >>> dill.detect.badobjects(your_bad_object, depth=1) >>> ... 

If you absolutely wanted this, you could use dill badobjects (or one of the other detection functions) to recursively dive into the object reference chain and push non-decomposable objects instead of naming them at each depth, as described above.

In addition, objgraph is a fairly convenient addition to the test suite.

 >>> # visualize the references in your bad objects >>> objgraph.show_refs(your_bad_object, filename='your_bad_object.png') 

Answer 2

I did it, it does the trick for me a lot of time ... I will update it when I find something completely error-free

It creates a bunch of fingerprints and then throws an exception if there is one that needs to be created so that you can see which part of the object hierarchy is causing the problem.

 def test_pickle(xThing,lTested = []): import pickle if id(xThing) in lTested: return lTested sType = type(xThing).__name__ print('Testing {0}...'.format(sType)) if sType in ['type','int','str']: print('...too easy') return lTested if sType == 'dict': print('...testing members') for k in xThing: lTested = Pickalable.test_pickle(xThing[k],lTested) print('...tested members') return lTested if sType == 'list': print('...testing members') for x in xThing: lTested = Pickalable.test_pickle(x) print('...tested members') return lTested lTested.append(id(xThing)) oClass = type(xThing) for s in dir(xThing): if s.startswith('_'): print('...skipping *private* thingy') continue #if it is an attribute: Skip it try: xClassAttribute = oClass.__getattribute__(oClass,s) except AttributeError: pass else: if type(xClassAttribute).__name__ == 'property': print('...skipping property') continue xAttribute = xThing.__getattribute__(s) print('Testing {0}.{1} of type {2}'.format(sType,s,type(xAttribute).__name__)) #if it is a function make sure it is stuck to the class... if type(xAttribute).__name__ == 'function': raise Exception('ERROR: found a function') if type(xAttribute).__name__ == 'method': print('...skipping method') continue if type(xAttribute).__name__ == 'HtmlElement': continue if type(xAttribute) == dict: print('...testing dict values for {0}.{1}'.format(sType,s)) for k in xAttribute: lTested = Pickalable.test_pickle(xAttribute[k]) continue print('...finished testing dict values for {0}.{1}'.format(sType,s)) try: oIter = xAttribute.__iter__() except AttributeError: pass except AssertionError: pass #lxml elements do this else: print('...testing iter values for {0}.{1} of type {2}'.format(sType,s,type(xAttribute).__name__)) for x in xAttribute: lTested = Pickalable.test_pickle(x,lTested) print('...finished testing iter values for {0}.{1}'.format(sType,s)) try: xAttribute.__dict__ except AttributeError: pass else: #this attribute should be explored seperately... lTested = Pickalable.test_pickle(xAttribute,lTested) continue pickle.dumps(xAttribute) print('Testing {0} as complete object'.format(sType)) pickle.dumps(xThing) return lTested