Unmodified C++ object after calling a python callback

[benEnsta]

Issue description

Hi,

I have a C++ code which calls a python function. This function modifies an object passed as its argument. When this object has been instantiated from python, modifications done by the python function are available to the c++ side.
However, an object, directly instantiated from the C++ side, is not modified after the function call.

The following example illustrates this behavior. With the function example_NOK, the object o1 is unmodified after the python function call. This is not the case with example_OK.

Is there a way to fix this issue and have my C++ object modified ?
Thanks for your help.

Reproducible example code

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <string>

namespace py = pybind11;

// custom type
class MyType {
public:
  MyType(int val) :val(val){}
  int val;
};

void example_OK(std::function<void(MyType&)> f, MyType& o){
  py::print("[example_OK] initial o = ", o);
  f(o);
  py::print("[example_OK] final o = ", o);
}

void example_NOK(std::function<void(MyType&)> f){
  MyType o1(10);
  py::print("[example_NOK] initial o = ", o1);
  f(o1);
  py::print("[example_NOK] final o = ", o1);
}

PYBIND11_MODULE(cmake_example, m) {
    py::class_<MyType>(m, "MyType")
      .def(py::init<int>())
      .def("__str__", [](MyType& o){return std::to_string(o.val);})
      .def_readwrite("val", &MyType::val)
    ;

    m.def("example_NOK", &example_NOK);
    m.def("example_OK", &example_OK);
}

python code:

import cmake_example as m

def f(o):
    o.val = -3

t = m.MyType(10)
m.example_OK(f, t)
m.example_NOK(f)

Ìt produces the following output

[example_OK] initial o =  10
[example_OK] final o =  -3
[example_NOK] initial o =  10
[example_NOK] final o =  10

In both case, we expected that the final value of the object was -3.

[EricCousineau-TRI]

Do you know if the code works if you change your function signature to std::function<void (MyType*)>, and pass it a pointer?

I may have ran into a similar issue where pybind was copying the object rather than referencing it (but got a compile-time error because the type was non-copyable).
Here is where I encountered this, and the shim I put in place:
https://github.com/RobotLocomotion/drake/blob/0215e565aa38aaa9271d7757174a1c5ec61bcd6c/bindings/pydrake/systems/framework_py.cc#L102

If this is indeed the issue, then the stop gap would be to write either a specific shim (like what I did) or a general shim where you effectively want to specify py::return_value_policy::reference rather than py::return_value_policy::automatic (which the docs say is ...::copy for lvalue references). (I may try this out shortly, will let you know if I do.)

A longer-term solution might be to have some way to say that you want lvalue arguments to default to reference if a copy is not necessary.