Make it possible to provide scope to/store scope of Python function

[ingomueller-net]

Issue description

A lambda function constructed with a certain scope (local and global) from within py::eval cannot be called, as the scope is unavailable when the function is called.

Reproducible example code

#include <pybind11/embed.h>

namespace py = pybind11;

int main(int, char**) {
    py::scoped_interpreter guard{};
    auto global = py::dict(py::module::import("__main__").attr("__dict__"));
    auto local = py::dict();
    local["x"] = 42;
    py::function f = py::eval("lambda: x", global, local);
    py::print("defined f");
    f();
    py::print("called f");
}

Output:

defined f
terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  NameError: name 'x' is not defined

Initial discussion

The problem is that the local variable a is not available when f is called; in fact, f does not seem to have a scope.

Note that the following works in Python:

def make_f():
    a=42
    return eval('lambda: a')
a=123
print(make_f()())   # outputs 123

I have no idea how, but it would be great if there was a way to provide a scope to py::functions to make above example work.

[ingomueller-net]

I tried many things and nothing worked. I must be missing some fundamental understanding of what locals and globals are.

One thing worked, though: making x global, i.e., saying globals["x"] = 42, or, more generally:

global.attr("update")(local);

I'd still be glad to hear why this is happening, though.

[eacousineau]

I think the core of this is due to the, uh, interesting nuance of how Python's scoping for globals locals works with eval + lambda:
https://stackoverflow.com/questions/50579541/why-lambda-in-eval-function-cant-closure-the-variables-in-the-user-defined-loc

Modifying your above code corroborates this:
https://github.com/eacousineau/repro/blob/3852e054da9489b132852e34898958e7b51706c0/python/lambda_eval_scope.py#L19

All in all, it seems that if you want to bind variables to your lambda, consider passing them with globals, as you're already doing.

Other alternatives are to implement your lambda using py::cpp_function, or using py::exec to assign to a local variable, which you can then retrieve. Examples:

// With cpp_function
py::function f = py::cpp_function([]() { return 42; });

// With exec
py::dict globals;
globals.attr("update")(py::globals());
globals["x"] = 42;
py::dict locals;
py::exec("f = lambda: x", globals, locals);
py::function f = locals["f"];

[ingomueller-net]

Thanks for the explanation; in particular, the SO post was helpful. For my use case, I think that passing the local variables as globals is the best solution, but knowing alternatives is great as well!

[EricCousineau-TRI]

Had forgotten this stuff, so I wrote a quick unittest example in #2743.

[YannickJadoul]

Seeing this now, after seeing #2743 (thanks, @EricCousineau-TRI!)

Note that the following works in Python:

def make_f():
    a=42
    return eval('lambda: a')
a=123
print(make_f()())   # outputs 123

It should be noted there that make_f()() also doesn't return the value of the local variable a = 42, but of the global one. So at least, pybind11 is consistent with the reasonably counterintuivitive behavior from Python.