Description
This has been distilled from a crash we found at Google while upgrading to Python 3.11. It's a complicated setup and I have only been able to partially reproduce the issue, but I believe my solution is correct.
When extension types set Py_TPFLAGS_HAVE_GC
they should call PyObject_GC_Untrack()
before starting destruction of the object so that the GC module doesn't see partially destructed objects if it happens to trigger during it. Python 3.11 started warning when types didn't do this correctly:
Lines 2398 to 2407 in d78c872
There's two subtle issues here: one is that Py_DECREF()
, and thus object deallocation, can be called with a pending exception (it's not unreasonable or even uncommon to do so). The PyErr_WarnExplicitFormat
call, however, may (and likely will) call other Python code, which will trigger an assertion that no exception is pending, usually this one:
Lines 4784 to 4785 in d78c872
The code should store any pending exception before calling anything that might care about the exception.
The other subtle issue is that the code warns before untracking the object. The warning (which can call arbitrary Python code via warnings.showwarning
) can easily trigger a GC run, which then turns up seeing the still-erroneously-tracked object in an invalid state. The act of warning about a potential issue is triggering the issue (but only with Py_DEBUG
defined). As far as I can see there's no reason not to untrack before raising the warning, which would avoid the problem. I have a PR that fixes both these issues.
Linked PRs
- gh-111777: Fix assertion errors on incorrectly still-tracked GC object destruction #111778
- [3.12] gh-111777: Fix assertion errors on incorrectly still-tracked GC object destruction (GH-111778) #111989
- [3.11] gh-111777: Fix assertion errors on incorrectly still-tracked GC object destruction (GH-111778) #111990