[BUG] Crash invoking Python function which fails with RecursionError

[RonAvitzur]

Issue description

With SymPy 1.6.1 and Python 3.8 embedded in C++, solveset(1/sqrt(x)) crashes.

solveset(1/sqrt(x)) fails reporting RecursionError: maximum recursion depth exceeded in the interactive Python interpreter. With embedded Python, it crashes the embedding app with an address error.

Reproducible example code

SString* pythonDirectory = [[NSBundle mainBundle] pathForResource:@"Python" ofType:nil];
Py_SetPythonHome(Py_DecodeLocale([pythonDirectory UTF8String], NULL));

py::initialize_interpreter();
py::module gSymPy = py::module::import("sympy");
py::object parse_expr = gSymPy.attr("parse_expr");
py::object solveset = gSymPy.attr("solveset");
try {
   py::object result = solveset(parse_expr("1/sqrt(x)"))
   }
catch (...) { }

crashes the embedding application with an address error 4084 stack frames deep.

Is there a way to handle this gracefully in the embedding app without fixing the underlying SymPy bug?
(#sympy/sympy#20097)

#0 0x00000001005ae49e in vgetargs1_impl at /Python/getargs.c:257
#1 0x00000001005ae2a1 in vgetargs1 [inlined] at /Python/getargs.c:434
#2 0x00000001005ae253 in PyArg_ParseTuple at /Python/getargs.c:129
#3 0x000000010050341f in super_init at /Objects/typeobject.c:7823
#4 0x0000000100500a94 in type_call at /Objects/typeobject.c:991
#5 0x00000001004a9096 in _PyObject_MakeTpCall at /Objects/call.c:159
#6 0x000000010059002c in call_function ()
#7 0x000000010058cdcd in _PyEval_EvalFrameDefault at /Python/ceval.c:3500
#8 0x0000000100591070 in PyEval_EvalFrameEx [inlined] at /Python/ceval.c:741
#9 0x0000000100591053 in _PyEval_EvalCodeWithName at /Python/ceval.c:4298
#10 0x00000001004a9be3 in _PyFunction_Vectorcall at /Objects/call.c:435
#11 0x0000000100503dfc in call_unbound_noarg [inlined] ()
#12 0x0000000100503dcd in slot_tp_hash at /Objects/typeobject.c:6479
#13 0x00000001004fbcea in tuplehash at /Objects/tupleobject.c:375
#14 0x0000000100630a68 in bounded_lru_cache_wrapper at /Modules/_functoolsmodule.c:939
#15 0x00000001004a954b in PyObject_Call at /Objects/call.c:245
#16 0x000000010058d00b in do_call_core [inlined] at /Python/ceval.c:5010
#17 0x000000010058cfdf in _PyEval_EvalFrameDefault at /Python/ceval.c:3559
...
#4069 0x000000010058cdcd in _PyEval_EvalFrameDefault at /Python/ceval.c:3500
#4070 0x0000000100591070 in PyEval_EvalFrameEx [inlined] at /Python/ceval.c:741
#4071 0x0000000100591053 in _PyEval_EvalCodeWithName at /Python/ceval.c:4298
#4072 0x00000001004a9be3 in _PyFunction_Vectorcall at /Objects/call.c:435
#4073 0x000000010058ff02 in _PyObject_Vectorcall [inlined] at /Include/cpython/abstract.h:127
#4074 0x000000010058fef6 in call_function at /Python/ceval.c:4963
#4075 0x000000010058ce6b in _PyEval_EvalFrameDefault at /Python/ceval.c:3515
#4076 0x0000000100591070 in PyEval_EvalFrameEx [inlined] at /Python/ceval.c:741
#4077 0x0000000100591053 in _PyEval_EvalCodeWithName at /Python/ceval.c:4298
#4078 0x00000001004a9be3 in _PyFunction_Vectorcall at /Objects/call.c:435
#4079 0x00000001004a93ec in PyVectorcall_Call at /Objects/call.c:199
#4080 0x00000001001a1dca in pybind11::detail::unpacking_collector<(pybind11::return_value_policy)1>::call(_object*) const at /pybind11/cast.h:2074
#4081 0x000000010017772d in pybind11::object pybind11::detail::object_apipybind11::handle::operator()<(pybind11::return_value_policy)1, pybind11::object, pybind11::object, pybind11::arg_v>(pybind11::object&&, pybind11::object&&, pybind11::arg_v&&) const at /pybind11/cast.h:2196

[bstaletic]

First, you didn't finalize the interpreter. Second, you didn't declare parse_expr. After fixing those two, your snippet looks like this:

#include <pybind11/embed.h>
namespace py = pybind11;
int main() {
	py::scoped_interpreter g;
	py::module gSymPy   = py::module::import("sympy");
	py::object gParse   = gSymPy.attr("parse_expr");
	py::object solveset = gSymPy.attr("solveset");
	py::object parse_expr = gSymPy.attr("parse_expr");
	py::object result = solveset(parse_expr("1/sqrt(x)"));
}

The last line raises RecursionError, that gets rethrown as py::error_already_set. Since py::error_already_set isn't caught anywhere, we should expect a crash.

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  RecursionError: maximum recursion depth exceeded while calling a Python object

At:
  /usr/lib/python3.8/site-packages/sympy/core/basic.py(2017): _preorder_traversal

zsh: abort      ./a.out

The above is with pybind11 2.5.0, python 3.8.5 and sympy 1.6.2.

However, I can see some objective-c in your snippet, which I know nothing about.

[RonAvitzur]

Right, thank you. I left out the try/catch portion of my source:

    try {
...
        }
    catch (...) { }

I seeing the crash on the call without seeing any C++ exceptions thrown here. I wonder what is different in my embedded configuration.

[bstaletic]

https://pybind11.readthedocs.io/en/master/advanced/exceptions.html#handling-exceptions-from-python-in-c

All python exceptions get translated to py::error_already_set. If there are no further questions, feel free to close this.

[RonAvitzur]

On further investigation, the one additional aspect in the embedding app which I had left out mistakenly thinking it irrelevant is that the app creates a pthread dedicated to Python which always holds the GIL and processes a command queue. On macOS here, pthread_create defaults to a 512K stack size for the thread, which is insufficient. Increases the thread's stack size to 2MB eliminated the crash so the exception now appears and is handled gracefully. Thank you for your patience.