test_opcache fails randomly (failure or crash)

[vstinner]

On a Free Threaded build:

$ ./configure --with-pydebug --disable-gil
$ make
$ ./python -m test test_opcache --forever -v

Error 1 (easy to reproduce):

FAIL: test_load_global_module (test.test_opcache.TestRacesDoNotCrash.test_load_global_module)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_opcache.py", line 1048, in test_load_global_module
    self.assert_races_do_not_crash(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        opname, get_items, read, write, check_items=True
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/python/main/Lib/test/test_opcache.py", line 590, in assert_races_do_not_crash
    self.assert_specialized(item, opname)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/test/test_opcache.py", line 23, in assert_specialized
    self.assertIn(opname, opnames)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: 'LOAD_GLOBAL_MODULE' not found in {'LOAD_GLOBAL', 'RESUME_CHECK', 'RETURN_VALUE'}

Error 2 (crash, harder to reproduce):

test_load_attr_method_with_values (test.test_opcache.TestRacesDoNotCrash.test_load_attr_method_with_values) ...

Fatal Python error: Segmentation fault

<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/home/vstinner/python/main/Lib/test/test_opcache.py", line 913 in write
  File "/home/vstinner/python/main/Lib/threading.py", line 1021 in run
  File "/home/vstinner/python/main/Lib/threading.py", line 1079 in _bootstrap_inner
  File "/home/vstinner/python/main/Lib/threading.py", line 1041 in _bootstrap

Current thread's C stack trace (most recent call first):
  Binary file "./python", at _Py_DumpStack+0x31 [0x6715c7]
  Binary file "./python" [0x684547]
  Binary file "./python" [0x6847bd]
  Binary file "/lib64/libc.so.6", at +0x19df0 [0x7f95d056bdf0]
  Binary file "./python", at _Py_Dealloc+0x18 [0x4e7a28]
  Binary file "./python", at _Py_MergeZeroLocalRefcount+0x58 [0x4e7b3e]
  Binary file "./python" [0x50f410]
  Binary file "./python" [0x51db0a]
  Binary file "./python", at _PyObject_GenericSetAttrWithDict+0xa3 [0x4e9e77]
  Binary file "./python", at PyObject_GenericSetAttr+0xe [0x4ea0dd]
  Binary file "./python", at PyObject_SetAttr+0x60 [0x4e8f5f]
  Binary file "./python", at PyObject_DelAttr+0xe [0x4e92fb]
  Binary file "./python", at _PyEval_EvalFrameDefault+0xed7b [0x5ce860]
  (...)
  <truncated rest of calls>

Extension modules: _testinternalcapi (total: 1)

cc @colesbury @nascheme

Linked PRs

• gh-132942: Fix races in type lookup cache #133032
• [3.13] gh-132942: Fix races in type lookup cache #133114

[Eclips4]

FYI, the first one failure is known: #122320

[colesbury]

Yeah, I've seen crashes in test_load_attr_method_with_values and test_binary_subscr_getitem. The issue appears to be related to the MRO cache and not specialization. I haven't been able to get a rr recording yet unfortunately, which makes finding the root cause more difficult.

[vstinner]

FYI, the first one failure is known: #122320

test_load_global_module() was fixed recently by #132961. The test is now skipped when run multiple times in a row, instead of failing:

test_load_global_module (test.test_opcache.TestRacesDoNotCrash.test_load_global_module) ... skipped 'Low on dict key versions'