generator throw delivered to incorrect generator under trace

[graingert]

Bug report

With the following demo code, a subgenerator yield from from a parent generator main(), that catches exceptions and returns "good":

class Inner:
    def send(self, v):
        return None

    def __iter__(self):
        return self

    def __next__(self):
        return self.send(None)

    def throw(self, *exc_info):
        raise StopIteration("good")


def main():
    return (yield from Inner())

def run(coro):
    coro.send(None)
    try:
        coro.throw(Exception)
    except StopIteration as e:
        print(e.value)


run(main())

when run with python -m trace, the exception is delivered to the outer generator main() instead of being suppressed

graingert@conscientious  ~/projects/cpython   main  ./python -m trace --count -C . demo.py           
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/graingert/projects/cpython/Lib/trace.py", line 741, in <module>
    main()
  File "/home/graingert/projects/cpython/Lib/trace.py", line 729, in main
    t.runctx(code, globs, globs)
  File "/home/graingert/projects/cpython/Lib/trace.py", line 451, in runctx
    exec(cmd, globals, locals)
  File "demo.py", line 26, in <module>
    run(main())
  File "demo.py", line 21, in run
    coro.throw(Exception)
  File "demo.py", line 16, in main
    return (yield from Inner())
            ^^^^^^^^^^^^^^^^^^
Exception
 ✘  graingert@conscientious  ~/projects/cpython   main ± ./python demo.py                      
good

the problem also occurs with the equivalent generator syntax, eg:

def Inner():
    try:
        while True:
            yield None
    except:
        return "good"

Your environment

• CPython versions tested on: c05c31d 3.12b1
• Operating system and architecture: Ubuntu 22.04 x86_64

see also

urllib3/urllib3#3049 (comment)
nedbat/coveragepy#1635

Linked PRs

• GH-105162: Account for INSTRUMENTED_RESUME in gen.close/throw. #105187
• [3.12] GH-105162: Account for INSTRUMENTED_RESUME in gen.close/throw. (GH-105187) #105378

[graingert]

@markshannon bisected to 411b169

[brandtbucher]

It might be related to the fact that YIELD_VALUE cannot raise...

cpython/Python/bytecodes.c

Lines 929 to 932 in a99b9d9

	inst(YIELD_VALUE, (retval -- unused)) {
	// NOTE: It's important that YIELD_VALUE never raises an exception!
	// The compiler treats any exception raised here as a failed close()
	// or throw() call.

...but INSTRUMENTED_YIELD_VALUE can:

cpython/Python/bytecodes.c

Lines 910 to 918 in a99b9d9

	inst(INSTRUMENTED_YIELD_VALUE, (retval -- unused)) {
	assert(frame != &entry_frame);
	PyGenObject *gen = _PyFrame_GetGenerator(frame);
	gen->gi_frame_state = FRAME_SUSPENDED;
	_PyFrame_SetStackPointer(frame, stack_pointer - 1);
	int err = _Py_call_instrumentation_arg(
	tstate, PY_MONITORING_EVENT_PY_YIELD,
	frame, next_instr-1, retval);
	if (err) goto error;

That seems incorrect.

[brandtbucher]

This is a little easier for me to follow:

def inner():
    try:
        yield
    except Exception:
        print("caught by inner")
        yield

def outer():
    try:
        yield from inner()
    except Exception:
        print("caught by outer")
        yield

gen = outer()
gen.send(None)
gen.throw(Exception)

brandtbucher@faster-cpython:~/cpython$ ./python bug.py
caught by inner
brandtbucher@faster-cpython:~/cpython$ ./python -m trace -c bug.py
caught by outer

[gaogaotiantian]

Will the actual exception in the trace function be thrown away if we simply remove the error check in instrumented version?

[brandtbucher]

Will the actual exception in the trace function be thrown away if we simply remove the error check in instrumented version?

No, we would need to explicitly clear it.

[Yhg1s]

Is there anything left to do for this issue?

[brandtbucher]

Nope!