Cython Pickling in Package "not found as" Error

Question

I have problems sorting the Cython class, but only when it is defined inside the package. This problem was noted earlier on the Internet , but they did not indicate how it was resolved. There are two components here: Cython etching using the __reduce__ method and package errors.

Pinton success cython

First I will show how this works without part of the package. This example is working correctly.

Cython file

My Cython reudce.pyx file:

 cdef class Foo(object): cdef int n def __init__(self, n): self.n = n def __reduce__(self): return Foo, (self.n,) 

Setup file

This can be compiled using setup.py :

 from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {'build_ext': build_ext}, ext_modules = [Extension("reduce", ["reduce.pyx"])] ) 

by executing python setup.py build && cp build/lib*/reduce.so .

Test script

The test script is called test_reduce.py and has the following form:

 import reduce import pickle f = reduce.Foo(4) print pickle.dumps(f) 

Running python test_reduce.py works fine.

Cython Spray on Packaging Failure

However, as soon as reduce.pyx is placed in the package, an error occurs.

Package creation

To reproduce this, first create a package called bar .

 mkdir bar mv reduce.so bar echo "from reduce import Foo" > bar/__init__.py 

Test script

Modify the test_reduce.py file as follows:

 import bar import pickle f = bar.Foo(4) print pickle.dumps(f) 

Error message

Running python test_reduce.py gives the following error:

 File "/usr/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python2.7/pickle.py", line 748, in save_global (obj, module, name)) pickle.PicklingError: Can't pickle <type 'reduce.Foo'>: it not found as reduce.Foo 

There is an error of errors that all turn into PicklingError in pickle.py . Looking at this code, a specific error occurs:

 ImportError: No module named reduce 

Welfare test

To check if there is any area or other problem, if I run the steps that the brine module should execute, everything works:

 f = bar.Foo(4) call, args = f.__reduce__() print call(*args) 

So what is going on here ?!

Answer 1

The problem was the build script. The Pickle module uses the __module__ attribute of the function / class to etch. This __module__ attribute comes from the first argument to the Extension() constructor in the setup.py script. Since I defined my constructor as Extension('reduce', ['reduce.pyx']) , the __module__ reduce attribute. It should be bar/reduce , although now it is in the package.

Creating setup.py looks like this:

 from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {'build_ext': build_ext}, ext_modules = [Extension('bar/reduce', ['reduce.pyx'])] ) 

solves the problem.

Answer 2

EDIT

maybe something like this is possible: Foo.__module__ = 'bar.reduce' in bar/__init__.py

Or you can use the copyreg module. Here are some snippets from copyreg code:

 """Helper to provide extensibility for pickle. This is only useful to add pickle support for extension types defined in C, not for instances of user-defined classes. """ def pickle(ob_type, pickle_function, constructor_ob=None): # ... # Example: provide pickling support for complex numbers. try: complex except NameError: pass else: def pickle_complex(c): return complex, (c.real, c.imag) pickle(complex, pickle_complex, complex) 

OLD VERSION

Pickle does this:

  try: __import__(module, level=0) mod = sys.modules[module] klass = getattr(mod, name) except (ImportError, KeyError, AttributeError): raise PicklingError( "Can't pickle %r: it not found as %s.%s" % (obj, module, name)) 

Can you check which line is not working?

When pickle piclkes classes and functions, such as module.function , it re-confirms that it is the correct function:

 # module.py def function(): # 1 pass a = function def function(): # 2 pass 

Function 1 in a cannot be pickled, but function 2 can be pickled because it is in a module under __name__ , which has a “function”.

So, in your case, pickle does not find the same Foo class in the reduce module, which is passed as an argument. The Foo argument claims to be found in the reduce module.