Test failure: System.Collections.Concurrent.Tests

[BruceForstall]

pipelines: runtime-coreclr libraries-pgo
net9.0-linux-Release-x64-CoreCLR_checked-jitosr_stress_random-Ubuntu.2204.Amd64.Open

https://dev.azure.com/dnceng-public/public/_build/results?buildId=417331&view=ms.vss-test-web.build-test-results-tab&runId=9140590&paneView=debug&resultId=200189

DOTNET_TC_QuickJitForLoops=1
DOTNET_EnableCrashReport=1
DOTNET_TieredCompilation=1
DOTNET_JitRandomOnStackReplacement=15
DOTNET_TC_OnStackReplacement=1
DOTNET_OSR_HitLimit=2
DOTNET_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
DOTNET_DbgEnableMiniDump=1
DOTNET_TC_OnStackReplacement_InitialCounter=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/B52A0A17/p
----- start Sun Sep 24 04:41:24 PM UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/B52A0A17/p/dotnet exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/B52A0A17/w/B3F009C2/e /datadisks/disk1/work/B52A0A17/w/B3F009C2/e
  Discovering: System.Collections.Concurrent.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Concurrent.Tests (found 897 of 905 test cases)
  Starting:    System.Collections.Concurrent.Tests (parallel test collections = on, max threads = 2)
[createdump] Gathering state for process 3742 dotnet
[createdump] Crashing thread 0ef4 signal 11 (000b)
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.3742.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.3742.dmp
[createdump] Written 341209088 bytes (83303 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 776ms
./RunTests.sh: line 204:  3742 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/B52A0A17/w/B3F009C2/e
----- end Sun Sep 24 04:41:47 PM UTC 2023 ----- exit code 139 ----------------------------------------------------------

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

net9.0-linux-Release-x64-CoreCLR_checked-jitosr_stress_random-Ubuntu.2204.Amd64.Open

https://dev.azure.com/dnceng-public/public/_build/results?buildId=417331&view=ms.vss-test-web.build-test-results-tab&runId=9140590&paneView=debug&resultId=200189

DOTNET_TC_QuickJitForLoops=1
DOTNET_EnableCrashReport=1
DOTNET_TieredCompilation=1
DOTNET_JitRandomOnStackReplacement=15
DOTNET_TC_OnStackReplacement=1
DOTNET_OSR_HitLimit=2
DOTNET_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
DOTNET_DbgEnableMiniDump=1
DOTNET_TC_OnStackReplacement_InitialCounter=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/B52A0A17/p
----- start Sun Sep 24 04:41:24 PM UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/B52A0A17/p/dotnet exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/B52A0A17/w/B3F009C2/e /datadisks/disk1/work/B52A0A17/w/B3F009C2/e
  Discovering: System.Collections.Concurrent.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Concurrent.Tests (found 897 of 905 test cases)
  Starting:    System.Collections.Concurrent.Tests (parallel test collections = on, max threads = 2)
[createdump] Gathering state for process 3742 dotnet
[createdump] Crashing thread 0ef4 signal 11 (000b)
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.3742.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.3742.dmp
[createdump] Written 341209088 bytes (83303 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 776ms
./RunTests.sh: line 204:  3742 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/B52A0A17/w/B3F009C2/e
----- end Sun Sep 24 04:41:47 PM UTC 2023 ----- exit code 139 ----------------------------------------------------------

Author:	BruceForstall
Assignees:	AndyAyersMS
Labels:	JitStress, arch-x64, area-CodeGen-coreclr, blocking-clean-ci-optional
Milestone:	9.0.0

[AndyAyersMS]

There is a similar looking failure here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=480142&view=results

So, will try and repro...

[AndyAyersMS]

No luck so far, have done several thousand runs (on WSL2) without a problem.

I downloaded the core dump from the 11/25 failure via runfo but wasn't able to do much with it. It looks like an AV in managed code, but LLDB+SOS seems unable to find libcoreclr.so so I have no idea what method was running.

[amanasifkhalid]

There was another AV on Linux x64 from a run on Saturday (failure log), though for System.Text.Encodings.Web.Tests. However, I haven't seen any System.Collections.Concurrent.Tests failures in recent runs of this pipeline. Is there anything else we can try with this issue in the meantime?

[BruceForstall]

Failed again:

net9.0-linux-Release-x64-CoreCLR_checked-jitosr_stress_random-Ubuntu.2204.Amd64.Open

https://dev.azure.com/dnceng-public/public/_build/results?buildId=511426&view=ms.vss-test-web.build-test-results-tab&runId=11969646&paneView=debug

DOTNET_TC_QuickJitForLoops=1
DOTNET_EnableCrashReport=1
DOTNET_TieredCompilation=1
DOTNET_JitRandomOnStackReplacement=15
DOTNET_TC_OnStackReplacement=1
DOTNET_OSR_HitLimit=2
DOTNET_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
DOTNET_DbgEnableMiniDump=1
DOTNET_TC_OnStackReplacement_InitialCounter=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/A8DB0929/p
========================= Begin custom configuration settings ==============================
export __TestArchitecture=x64
export __IsXUnitLogCheckerSupported=1
========================== End custom configuration settings ===============================
----- start Sun Dec 31 04:53:39 PM UTC 2023 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/A8DB0929/p/dotnet exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/A8DB0929/w/B69E09EF/e /datadisks/disk1/work/A8DB0929/w/B69E09EF/e
  Discovering: System.Collections.Concurrent.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Collections.Concurrent.Tests (found 898 of 906 test cases)
  Starting:    System.Collections.Concurrent.Tests (parallel test collections = on, max threads = 2)
[createdump] Gathering state for process 5301 dotnet
[createdump] Crashing thread 14c6 signal 11 (000b)
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.5301.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.5301.dmp
[createdump] Written 278876160 bytes (68085 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 653ms
./RunTests.sh: line 182:  5301 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Collections.Concurrent.Tests.runtimeconfig.json --depsfile System.Collections.Concurrent.Tests.deps.json xunit.console.dll System.Collections.Concurrent.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/A8DB0929/w/B69E09EF/e
----- end Sun Dec 31 04:53:57 PM UTC 2023 ----- exit code 139 ----------------------------------------------------------

[AndyAyersMS]

Seems likely this was the same issue as in #98292 and #101060.

Fix is #101709.