returning Python list of wrapped non-copyable type

[Xeverous]

Issue description

Returning a list of non copyable type raises an exception. This should not happen.

Reproducible example code

#include <pybind11/pybind11.h>

struct non_copyable
{
        non_copyable() = default;

        non_copyable(const non_copyable&) = delete;
        non_copyable& operator=(non_copyable) = delete;
        non_copyable(non_copyable&&) = default;
        non_copyable& operator=(non_copyable&&) = default;
};

PYBIND11_MODULE(test_mod, m)
{
        py::class_<non_copyable>(m, "NonCopyable")
                .def(py::init());
        m.def("get_list", []()
        {
                py::list list;
                list.append(non_copyable());
                list.append(non_copyable());
                return list;
        });
}

import test_mod

l = test_mod.get_list()
RuntimeError: return_value_policy = copy, but the object is non-copyable!

How can I return a list of objects that are not copyable? This looks like a bug to me.

[Xeverous]

It seems that the problem lies here:

pybind11::list::append() takes T&& and calls PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());

object_or_cast has 3 overloads:

pytypes.h, line 425:

inline handle object_or_cast(PyObject *ptr) { return ptr; }

cast.h, line 1667:

template <typename T, enable_if_t<!is_pyobject<T>::value, int>>
object object_or_cast(T &&o) { return pybind11::cast(std::forward<T>(o)); }

pytypes.h, line 419:

template <typename T, enable_if_t<is_pyobject<T>::value, int> = 0>
auto object_or_cast(T &&o) -> decltype(std::forward<T>(o)) { return std::forward<T>(o); }

T here is not a PyObject* and is not a wrapper Python type so it goes into second overload: return pybind11::cast(std::forward<T>(o));

pybind11:cast has 6 overloads:

• (const handle&)
• (const handle&)
• (object&&)
• (object&&)
• (object&&)
• (const T&, return_value_policy, handle)

T is not a handle or an object wrapper so it picks the last, the most generic overload. At this place T&& is changed to const T& which unnecessarily copies moveable object OR compiles to runtime cast error with non-copyable type.

// C++ type -> py::object
template <typename T, detail::enable_if_t<!detail::is_pyobject<T>::value, int> = 0>
object cast(const T &value, return_value_policy policy = return_value_policy::automatic_reference,
            handle parent = handle()) {
    if (policy == return_value_policy::automatic)
        policy = std::is_pointer<T>::value ? return_value_policy::take_ownership : return_value_policy::copy;
    else if (policy == return_value_policy::automatic_reference)
        policy = std::is_pointer<T>::value ? return_value_policy::reference : return_value_policy::copy;
    return reinterpret_steal<object>(detail::make_caster<T>::cast(value, policy, parent)); // calls type caster
}

type caster has multiple overloads, including T&& but at this call value is already const T&. I think the above snippet should not take T by const reference but a forwarding reference and then call detail::make_caster<T>::cast(std::forward<T>(value), policy, parent)

[bstaletic]

Thanks for the detailed report, but this is already fixed on master. #1260