test_cprofile crashed on main

[kumaraditya303]

Commit tested: 1834133

Reproducer:

./python -m test

Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7fe79c68cbb0
object refcount : 0
object type     : 0x55d1aa0d6f10
object type name: _lsprof.Profiler
object repr     : <refcnt 0 at 0x7fe79c68cbb0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007fe7a2694280 (most recent call first):
  Garbage-collecting
  File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
  File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
  File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
  File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
Aborted (core dumped)

Interestingly it does not crashes if run with multiple processes.

cc @pablogsal

[pablogsal]

Does this happen on 3.11?

[kumaraditya303]

3.11 does not seem affected, I tried running tests but haven't got any crashes yet.

[pablogsal]

I am not able to reproduce this on my laptop. Will try again on my Linux machine later today

[kumaraditya303]

I am able to reproduce on Linux with the current HEAD main.

[pablogsal]

Can you bisect it?

[pablogsal]

Cannot reproduce in my Linux box:

./python -m test
== CPython 3.12.0a0 (heads/main:23f6944c37, Jul 24 2022, 15:14:06) [GCC 12.1.0]
== Linux-5.18.13-arch1-1-x86_64-with-glibc2.35 little-endian
== cwd: /home/pablogsal/github/python/main/build/test_python_2216269æ
== CPU count: 36
== encodings: locale=UTF-8, FS=utf-8

 ........

== Tests result: SUCCESS ==

417 tests OK.

19 tests skipped:
    test_curses test_devpoll test_kqueue test_launcher test_msilib
    test_ossaudiodev test_smtpnet test_socketserver test_startfile
    test_tix test_tkinter test_ttk test_urllib2net test_urllibnet
    test_winconsoleio test_winreg test_winsound test_xmlrpc_net
    test_zipfile64

Total duration: 40 min 15 sec
Tests result: SUCCESS

[kumaraditya303]

I can reproduce on main HEAD 78eb3f7

Crash:

Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7ff199c0b8c0
object refcount : 0
object type     : 0x556cad2891e0
object type name: _lsprof.Profiler
object repr     : <refcnt 0 at 0x7ff199c0b8c0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ff19d4e6280 (most recent call first):
  Garbage-collecting
  File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
  File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
  File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
  File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
Aborted (core dumped)

Environment:

== CPython 3.12.0a0 (heads/main:78eb3f79ea, Jul 27 2022, 10:43:00) [GCC 9.4.0]
== Linux-5.4.0-1074-azure-x86_64-with-glibc2.31 little-endian
== cwd: /workspaces/cpython/build/test_python_4447æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8

Another things is that if you remove test_asyncio then it passes, my guess is that it has something to do with the code specialization triggered by test_asyncio as that test executes a lot of code.

cc @markshannon @brandtbucher

I'll bisect later.

[kumaraditya303]

Turns out the profiler's deallocator did not untracked the object leading to the gc crash.

See #95315