Smart pointers & ``py::class_`` ############################### The binding generator for classes, ``py::class_``, can be passed a template type that denotes a special *holder* type that is used to manage references to the object. If no such holder type template argument is given, the default for a type ``T`` is ``std::unique_ptr``. .. note:: A ``py::class_`` for a given C++ type ``T`` — and all its derived types — can only use a single holder type. .. _smart_holder: ``py::smart_holder`` ==================== Starting with pybind11v3, ``py::smart_holder`` is built into pybind11. It is the recommended ``py::class_`` holder for most situations. However, for backward compatibility it is **not** the default holder, and there are no plans to make it the default holder in the future. It is extremely easy to use the safer and more versatile ``py::smart_holder``: simply add ``py::smart_holder`` to ``py::class_``: * ``py::class_`` to * ``py::class_``. .. note:: A shorthand, ``py::classh``, is provided for ``py::class_``. The ``h`` in ``py::classh`` stands for **smart_holder** but is shortened for brevity, ensuring it has the same number of characters as ``py::class_``. This design choice facilitates easy experimentation with ``py::smart_holder`` without introducing distracting whitespace noise in diffs. The ``py::smart_holder`` functionality includes the following: * Support for **two-way** Python/C++ conversions for both ``std::unique_ptr`` and ``std::shared_ptr`` **simultaneously**. * Passing a Python object back to C++ via ``std::unique_ptr``, safely **disowning** the Python object. * Safely passing "trampoline" objects (objects with C++ virtual function overrides implemented in Python, see :ref:`overriding_virtuals`) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++: associated Python objects are automatically kept alive for the lifetime of the smart-pointer. * Full support for ``std::enable_shared_from_this`` (`cppreference `_). ``std::unique_ptr`` =================== This is the default ``py::class_`` holder and works as expected in most situations. However, handling base-and-derived classes involves a ``reinterpret_cast``, which is, strictly speaking, undefined behavior. Also note that the ``std::unique_ptr`` holder only supports passing a ``std::unique_ptr`` from C++ to Python, but not the other way around. For example, the following code works as expected with ``py::class_``: .. code-block:: cpp std::unique_ptr create_example() { return std::unique_ptr(new Example()); } .. code-block:: cpp m.def("create_example", &create_example); However, this will fail with ``py::class_`` (but works with ``py::class_``): .. code-block:: cpp void do_something_with_example(std::unique_ptr ex) { ... } .. note:: The ``reinterpret_cast`` mentioned above is `here `_. For completeness: The same cast is also applied to ``py::smart_holder``, but that is safe, because ``py::smart_holder`` is not templated. ``std::shared_ptr`` =================== It is possible to use ``std::shared_ptr`` as the holder, for example: .. code-block:: cpp py::class_ /* <- holder type */>(m, "Example"); Compared to using ``py::class_``, there are two noteworthy disadvantages: * Because a ``py::class_`` for a given C++ type ``T`` can only use a single holder type, ``std::unique_ptr`` cannot even be passed from C++ to Python. This will become apparent only at runtime, often through a segmentation fault. * Similar to the ``std::unique_ptr`` holder, the handling of base-and-derived classes involves a ``reinterpret_cast`` that has strictly speaking undefined behavior, although it works as expected in most situations. .. _smart_pointers: Custom smart pointers ===================== For custom smart pointers (e.g. ``c10::intrusive_ptr`` in pytorch), transparent conversions can be enabled using a macro invocation similar to the following. It must be declared at the top namespace level before any binding code: .. code-block:: cpp PYBIND11_DECLARE_HOLDER_TYPE(T, SmartPtr) The first argument of :func:`PYBIND11_DECLARE_HOLDER_TYPE` should be a placeholder name that is used as a template parameter of the second argument. Thus, feel free to use any identifier, but use it consistently on both sides; also, don't use the name of a type that already exists in your codebase. The macro also accepts a third optional boolean parameter that is set to false by default. Specify .. code-block:: cpp PYBIND11_DECLARE_HOLDER_TYPE(T, SmartPtr, true) if ``SmartPtr`` can always be initialized from a ``T*`` pointer without the risk of inconsistencies (such as multiple independent ``SmartPtr`` instances believing that they are the sole owner of the ``T*`` pointer). A common situation where ``true`` should be passed is when the ``T`` instances use *intrusive* reference counting. Please take a look at the :ref:`macro_notes` before using this feature. By default, pybind11 assumes that your custom smart pointer has a standard interface, i.e. provides a ``.get()`` member function to access the underlying raw pointer. If this is not the case, pybind11's ``holder_helper`` must be specialized: .. code-block:: cpp // Always needed for custom holder types PYBIND11_DECLARE_HOLDER_TYPE(T, SmartPtr) // Only needed if the type's `.get()` goes by another name namespace PYBIND11_NAMESPACE { namespace detail { template struct holder_helper> { // <-- specialization static const T *get(const SmartPtr &p) { return p.getPointer(); } }; }} The above specialization informs pybind11 that the custom ``SmartPtr`` class provides ``.get()`` functionality via ``.getPointer()``. .. note:: The two noteworthy disadvantages mentioned under the ``std::shared_ptr`` section apply similarly to custom smart pointer holders, but there is no established safe alternative in this case. .. seealso:: The file :file:`tests/test_smart_ptr.cpp` contains a complete example that demonstrates how to work with custom reference-counting holder types in more detail. Be careful not to accidentally undermine automatic lifetime management ====================================================================== ``py::class_``-wrapped objects automatically manage the lifetime of the wrapped C++ object, in collaboration with the chosen holder type. When wrapping C++ functions involving raw pointers, care needs to be taken to not inadvertently transfer ownership, resulting in multiple Python objects acting as owners, causing heap-use-after-free or double-free errors. For example: .. code-block:: cpp class Child { }; class Parent { public: Parent() : child(std::make_shared()) { } Child *get_child() { return child.get(); } /* DANGER */ private: std::shared_ptr child; }; PYBIND11_MODULE(example, m) { py::class_>(m, "Child"); py::class_>(m, "Parent") .def(py::init<>()) .def("get_child", &Parent::get_child); /* PROBLEM */ } The following Python code leads to undefined behavior, likely resulting in a segmentation fault. .. code-block:: python from example import Parent print(Parent().get_child()) Part of the ``/* PROBLEM */`` here is that pybind11 falls back to using ``return_value_policy::take_ownership`` as the default (see :ref:`return_value_policies`). The fact that the ``Child`` instance is already managed by ``std::shared_ptr`` is lost. Therefore pybind11 will create a second independent ``std::shared_ptr`` that also claims ownership of the pointer, eventually leading to heap-use-after-free or double-free errors. There are various ways to resolve this issue, either by changing the ``Child`` or ``Parent`` C++ implementations (e.g. using ``std::enable_shared_from_this`` as a base class for ``Child``, or adding a member function to ``Parent`` that returns ``std::shared_ptr``), or if that is not feasible, by using ``return_value_policy::reference_internal``. What is the best approach depends on the exact situation.