Boost :: python and weak_ptr: things disappear

Question

I would like to keep the object reference as weak_ptr. In pure C ++, the following works:

#include <iostream> #include <boost/shared_ptr.hpp> #include <boost/weak_ptr.hpp> using namespace std; using namespace boost; struct Empty { Empty(){} }; struct Store { weak_ptr<Empty> value; Store(){}; void setValue(shared_ptr<Empty> v) { cout << "storing " << v << endl; this->value = weak_ptr<Empty>(v); shared_ptr<Empty> v_ok = this->value.lock(); if (v_ok) { cout << "ok, v has been stored" << endl; } } shared_ptr<Empty> getValue() { shared_ptr<Empty> p = this->value.lock(); if (p) { cout << "stored value : " << p << endl; } else { cout << "there nothing here !" << endl; } return p; } }; int main() { shared_ptr<Empty> e(new Empty); shared_ptr<Store> st(new Store); st->setValue(e); st->getValue(); return 0; } 

Compiling and doing this will give you the following:

 %> ./a.out storing 0x8c6c008 ok, v has been stored stored value : 0x8c6c008 

Now, if I encapsulate this with boost python:

 #include <iostream> #include <boost/shared_ptr.hpp> #include <boost/python.hpp> #include <boost/weak_ptr.hpp> using namespace std; using namespace boost; using namespace boost::python; struct Empty { Empty(){} }; struct Store { weak_ptr<Empty> value; Store(){}; void setValue(shared_ptr<Empty> v) { cout << "storing " << v << endl; this->value = weak_ptr<Empty>(v); shared_ptr<Empty> v_ok = this->value.lock(); if (v_ok) { cout << "ok, v has been stored" << endl; } } shared_ptr<Empty> getValue() { shared_ptr<Empty> p = this->value.lock(); if (p) { cout << "stored value : " << p << endl; } else { cout << "there nothing here !" << endl; } return p; } }; BOOST_PYTHON_MODULE (test) { class_< Empty, shared_ptr<Empty> >("Empty"); class_< Store, shared_ptr<Store> >("Store") .def("get",&Store::getValue) .def("set",&Store::setValue); } 

and now a small python script to try it

 from test import * e = Empty() st = Store() st.set(e) st.get() 

... and the result ...

 storing 0x9eb2a18 ok, v has been stored there nothing here ! 

so, apparently, while I'm still in one method (setValue), there is no problem retrieving shared_ptr from Store :: value. But as soon as I get out of this context, there is nothing left!

How can it be? Is python passing a completely new (and useless) shared_ptr as an argument to setValue, which is then destroyed at the end of the call? I'm lost here.

Answer 1

This is very curious. I ruled out the possibility of sharing std and boost, and also checked a few health check checks, and as far as I can tell, this is what raises the python level for a common pointer that breaks it.

Constructors / destructors of trace objects, empty and storage lifetimes are managed as you expected (without copying).

It’s extremely interesting that shared_from_this continues to work even if weak_ptr<>.lock() doesn’t work, and in fact a new weak pointer works, created from a new shared pointer (from shared_from_this).

So this led me to a topic related to comments , it looks like something is pushing python towards deleter and reference counting that breaks a weak pointer.

Checking the shared pointers in the debugger is what we get:

When we call setValue , it looks like this:

 1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe720: { px = 0x100346900, pn = { pi_ = 0x100338dd0 } } > p *p.pn.pi_ $5 = (boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>) { <boost::detail::sp_counted_base> = { _vptr$sp_counted_base = 0x10061aa30, use_count_ = 2, weak_count_ = 2 }, members of boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>: ptr = 0x0, del = { owner = { m_p = 0x10049db90 } } } 

If we create a shared pointer using shared_from_this in the argument, it looks like this:

 1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe5e0: { px = 0x100346900, pn = { pi_ = 0x1003468e0 } } > p *p.pn.pi_ $4 = (boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >) { <boost::detail::sp_counted_base> = { _vptr$sp_counted_base = 0x10061b170, use_count_ = 2, weak_count_ = 2 }, members of boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >: ptr = 0x0, del = { initialized_ = true, storage_ = { data_ = "\000i4\000\001\000\000\000?h4\000\001\000\000", align_ = {<No data fields>} } } } 

Here's one note: the address of the shared counter is different : this is another shared instance of the pointer ... so we somehow created two different shared pointers to the same address. This is very bad, because we expect this to lead soon to double release.

However, it is not. (I really want to understand this further if anyone has any ideas?)

I don’t know why this is not so, to be honest (apparently there is something subtle here), but in any case all this points to a solution: we can use shared_from_this to create a common pointer, from the passed value, which we can use to create a weak pointer that really works.

So, to wrap, here is the fix:

 #include <iostream> #include <boost/shared_ptr.hpp> #include <boost/python.hpp> #include <boost/weak_ptr.hpp> #include <boost/enable_shared_from_this.hpp> namespace bp = boost::python; struct Empty: boost::enable_shared_from_this<Empty>{ }; struct Store { boost::weak_ptr<Empty> value; void setValue(boost::shared_ptr<Empty> const& v) { value = boost::weak_ptr<Empty>(v->shared_from_this()); boost::shared_ptr<Empty> v_ok = value.lock(); if (v_ok) { std::cout << "ok, v has been stored" << std::endl; } } boost::shared_ptr<Empty> getValue() { boost::shared_ptr<Empty> p = value.lock(); if (p) { std::cout << "stored value : " << p << std::endl; } else { std::cout << "there nothing here !" << std::endl; } return p; } }; BOOST_PYTHON_MODULE (libmylibinterface) { bp::class_< Empty, boost::shared_ptr<Empty> >("Empty",bp::init<>()) ; bp::class_< Store, boost::shared_ptr<Store> >("Store") .def("get",&Store::getValue) .def("set",&Store::setValue); } 

Answer 2

Another workaround is to pass a reference to the shared_ptr object, not the shared_ptr object. In this case, obviously, boost::python-Wrapper does not create this weird separate shared_ptr .

Btw. I will look at the type of the class pointed to by shared_ptr.pn.pi_, you will see that the "original" shared_ptr passed to Python contains a pointer to the sp_counted_impl_p<POINTEE_TYPE> object, and shared_ptr back from Python contains a pointer to sp_counted_impl_pd<void* , shared_ptr_deleter> object (see smart_ptr/detail/sp_counted_impl.hpp ) The pd variant does not contain a pointer to a pointer. I suspect this sp_counted_impl_pd object sp_counted_impl_pd somehow referencing the sp_counted_impl_p object. This explains why the destructor is not called when the shared_ptr reference count passed back from Python drops to zero. The fact that weak_ptr does not work in this case might just be a mistake then.?