Direct class dict modification through gc.get_referrers() leads to an invalid type cache and a segfault

[jstasiak]

Crash report

What happened?

Over at Eventlet an interesting crash has been discovered: eventlet/eventlet#864

I managed to narrow it down to something like this

# crash.py
import gc

class Cls:
    var = []

for ref in gc.get_referrers(Cls.var):
    del ref['var']

# Depending on the environment only a subset of these calls is needed
# to trigger the segfault:
gc.collect()
print(Cls.var)
gc.collect()
print(Cls.var)

It crashes with all currently supported Python versions (3.8-3.12) and with the current main branch:

% python3 crash.py
[]
zsh: segmentation fault  python3 crash.py

I'm not familiar with this code but I think the problem is caused by the type cache being unaware of the class' dictionary being modified. If I disable the type cache lookup in _PyType_Lookup like so

diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index ea29a38d74ae..c6e65593b454 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -4759,13 +4759,13 @@ _PyType_Lookup(PyTypeObject *type, PyObject *name)
     unsigned int h = MCACHE_HASH_METHOD(type, name);
     struct type_cache *cache = get_type_cache();
     struct type_cache_entry *entry = &cache->hashtable[h];
-    if (entry->version == type->tp_version_tag &&
-        entry->name == name) {
-        assert(_PyType_HasFeature(type, Py_TPFLAGS_VALID_VERSION_TAG));
-        OBJECT_STAT_INC_COND(type_cache_hits, !is_dunder_name(name));
-        OBJECT_STAT_INC_COND(type_cache_dunder_hits, is_dunder_name(name));
-        return entry->value;
-    }
+    // if (entry->version == type->tp_version_tag &&
+    //     entry->name == name) {
+    //     assert(_PyType_HasFeature(type, Py_TPFLAGS_VALID_VERSION_TAG));
+    //     OBJECT_STAT_INC_COND(type_cache_hits, !is_dunder_name(name));
+    //     OBJECT_STAT_INC_COND(type_cache_dunder_hits, is_dunder_name(name));
+    //     return entry->value;
+    // }
     OBJECT_STAT_INC_COND(type_cache_misses, !is_dunder_name(name));
     OBJECT_STAT_INC_COND(type_cache_dunder_misses, is_dunder_name(name));

the issue goes away:

% ~/projects/cpython/python.exe crash.py 
Traceback (most recent call last):
  File "/Users/user/crash.py", line 12, in <module>
    print(Cls.var)
          ^^^^^^^
AttributeError: type object 'Cls' has no attribute 'var'

I don't feel confident enough to produce a good fix for this at this stage.

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, 3.12, CPython main branch

Operating systems tested on:

Linux, macOS

Output from running 'python -VV' on the command line:

No response

[tipabu]

Can confirm. Running with gdb and -X dev off main (3aadb950) gets me the following stack:

[]

Program received signal SIGSEGV, Segmentation fault.
0x0000555555814208 in _PyType_HasFeature (feature=16384, type=0xdddddddddddddddd) at ./Include/internal/pycore_object.h:260
260         return ((type->tp_flags & feature) != 0);
(gdb) bt 10
#0  0x0000555555814208 in _PyType_HasFeature (feature=16384, type=0xdddddddddddddddd) at ./Include/internal/pycore_object.h:260
#1  _PyType_PreHeaderSize (tp=0xdddddddddddddddd) at ./Include/internal/pycore_object.h:453
#2  PyObject_GC_Del (op=0x7ffff778bb10) at Modules/gcmodule.c:2384
#3  0x0000555555678fd0 in _PyList_ClearFreeList (interp=interp@entry=0x555555aba978 <_PyRuntime+76952>) at Objects/listobject.c:130
#4  0x0000555555813626 in clear_freelists (interp=0x555555aba978 <_PyRuntime+76952>) at Modules/gcmodule.c:1075
#5  gc_collect_main (tstate=tstate@entry=0x555555b20420 <_PyRuntime+493376>, generation=<optimized out>, generation@entry=2, reason=reason@entry=_Py_GC_REASON_MANUAL)
    at Modules/gcmodule.c:1488
#6  0x0000555555813b12 in PyGC_Collect () at Modules/gcmodule.c:2136
#7  0x00005555557dd29e in Py_FinalizeEx () at Python/pylifecycle.c:1924
#8  0x00005555557e02dd in Py_FinalizeEx () at Python/pylifecycle.c:2045
#9  0x00005555558111f5 in Py_RunMain () at Modules/main.c:709
(More stack frames follow...)

---

With a similar crashing script

import gc
class A:
        attr = object()  # or some other thing that can get gc'ed
for r in gc.get_referrers(A.attr):
        r['attr'] = None
print(A.attr)

I get a slightly different stack

Program received signal SIGSEGV, Segmentation fault.
0x00005555556e7957 in _Py_type_getattro_impl (suppress_missing_attribute=0x0, name=0x7ffff782b560, type=0x555555c0d1d0) at Objects/typeobject.c:4876
4876            descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get;
(gdb) bt 10
#0  0x00005555556e7957 in _Py_type_getattro_impl (suppress_missing_attribute=0x0, name=0x7ffff782b560, type=0x555555c0d1d0) at Objects/typeobject.c:4876
#1  _Py_type_getattro (type=0x555555c0d1d0, name=0x7ffff782b560) at Objects/typeobject.c:4924
#2  0x00005555556a87f7 in PyObject_GetAttr (v=v@entry=0x555555c0d1d0, name=0x7ffff782b560) at Objects/object.c:1155
#3  0x00005555555d729b in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb7020, throwflag=<optimized out>) at Python/generated_cases.c.h:3441
#4  0x0000555555782aff in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb7020, tstate=0x555555b20420 <_PyRuntime+493376>) at ./Include/internal/pycore_ceval.h:115
#5  _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7ffff77d49b0, func=0x7ffff77bdc10, tstate=0x555555b20420 <_PyRuntime+493376>) at Python/ceval.c:1771
#6  PyEval_EvalCode (co=co@entry=0x7ffff78f1fc0, globals=globals@entry=0x7ffff77d49b0, locals=locals@entry=0x7ffff77d49b0) at Python/ceval.c:592
#7  0x00005555557e60e9 in run_eval_code_obj (locals=0x7ffff77d49b0, globals=0x7ffff77d49b0, co=0x7ffff78f1fc0, tstate=0x555555b20420 <_PyRuntime+493376>) at Python/pythonrun.c:1294
#8  run_mod (mod=<optimized out>, filename=filename@entry=0x7ffff7822920, globals=globals@entry=0x7ffff77d49b0, locals=locals@entry=0x7ffff77d49b0, flags=flags@entry=0x7fffffffe208, 
    arena=arena@entry=0x7ffff792c430, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1379
#9  0x00005555557e7f1c in pyrun_file (flags=0x7fffffffe208, closeit=1, locals=0x7ffff77d49b0, globals=0x7ffff77d49b0, start=257, filename=0x7ffff7822920, fp=0x555555b7de50)
    at Python/pythonrun.c:1215
(More stack frames follow...)

though it also seems to be resolved by disabling the type cache lookup.

[serhiy-storchaka]

It is a known issue, and the recommended solution -- does not do this.

[grigoriev-semyon]

It is a known issue, and the recommended solution -- does not do this.

I'm trying to make a PR. Can you give me a little hint on what needs to be done to fix this?
I think that I need to add cache clearing somewhere, but I don’t really understand where

[serhiy-storchaka]

It should be discussed on the eventlet tracker. There is nothing that can be done from Python side, except removing gc.get_referrers() and similar functions or removing the type cache.

[markshannon]

I don't see why this is an eventlet issue.
The above code is pure Python, with no reference to eventlet.

The immediate cause is that gc.get_referrers(Cls.var) exposes a mutable dict underlying the Cls.__dict__ mapping proxy.
The root cause is that such a dict exists at all.
This is an old bug, which we really should fix:

• There is a way to access an underlying mapping in MappingProxyType #88004