Skip to content

Segmentation fault while executing System.Net tests in linux x64 coreclr #83482

New issue
New issue
Closed
Closed
Segmentation fault while executing System.Net tests in linux x64 coreclr#83482
Labels
arch-x64area-System.Netarea-VM-coreclros-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtime
Milestone
8.0.0
@carlossanlop

Description

@carlossanlop
carlossanlop
opened on Mar 15, 2023

Found in main: #83356
And also found in 7.0: #83467

There is also a crash in these tests, but affecting mono, and also happens in both main and 7.0. They could be related: #83481

The output of all the 3 executions is very similar:

Output (expand) 
Console log: 'System.Net.Security.Unit.Tests' from job e1438f85-bd3c-4620-ae7b-ca686d95a426 workitem ee14f405-843d-442a-81cc-1dbd5bfc751f (redhat.7.amd64.open.svc) executed on machine a0009LP running Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo
+ ./RunTests.sh --runtime-path /mnt/work/AFB8097B/p
----- start Wed Mar 15 19:48:23 UTC 2023 =============== To repro directly: =====================================================
pushd .
/mnt/work/AFB8097B/p/dotnet exec --runtimeconfig System.Net.Security.Unit.Tests.runtimeconfig.json --depsfile System.Net.Security.Unit.Tests.deps.json xunit.console.dll System.Net.Security.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/mnt/work/AFB8097B/w/AEB109CE/e /mnt/work/AFB8097B/w/AEB109CE/e
  Discovering: System.Net.Security.Unit.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Unit.Tests (found 74 of 76 test cases)
  Starting:    System.Net.Security.Unit.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168: 24497 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Security.Unit.Tests.runtimeconfig.json --depsfile System.Net.Security.Unit.Tests.deps.json xunit.console.dll System.Net.Security.Unit.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/mnt/work/AFB8097B/w/AEB109CE/e
----- end Wed Mar 15 19:48:24 UTC 2023 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited
[    6.300543] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    6.300618] sd 3:0:1:0: Attached scsi generic sg1 type 0
[    6.300684] sr 2:0:0:0: Attached scsi generic sg2 type 5
[    6.390928] cryptd: max_cpu_qlen set to 1000
[    6.450462] XFS (sda2): Mounting V5 Filesystem
[    6.450493] XFS (dm-4): Mounting V5 Filesystem
[    6.450569] XFS (dm-0): Mounting V5 Filesystem
[    6.450600] XFS (dm-2): Mounting V5 Filesystem
[    6.450684] XFS (dm-3): Mounting V5 Filesystem
[    6.482819] XFS (dm-3): Ending clean mount
[    6.486027] XFS (sda2): Ending clean mount
[    6.616152] AVX2 version of gcm_enc/dec engaged.
[    6.616155] AES CTR mode by8 optimization enabled
[    6.619031] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    6.619177] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
[    6.670151] XFS (dm-0): Ending clean mount
[    6.809153] XFS (dm-2): Ending clean mount
[    7.060075] XFS (dm-4): Ending clean mount
[    7.080417] systemd-journald[539]: Received request to flush runtime journal from PID 1
[    7.222850] type=1305 audit(1678908232.101:4): audit_pid=751 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[    8.830299] UDF-fs: INFO Mounting volume 'UDF Volume', timestamp 2023/03/16 00:00 (1000)
[    9.071384] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    9.071391] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.950462] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   10.410842] hv_utils: KVP IC version 4.0
[   11.666556]  sdb: sdb1
[   12.641607] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[   18.061453] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   53.193928] hv_balloon: Max. dynamic memory size: 8192 MB
[   61.216830] SELinux: 2048 avtab hash slots, 112733 rules.
[   61.250698] SELinux: 2048 avtab hash slots, 112733 rules.
[   61.269664] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   61.269668] SELinux:  130 classes, 112733 rules
[   61.272716] SELinux:  Converting 1337 SID table entries...
[   68.824122] SELinux: 2048 avtab hash slots, 112735 rules.
[   68.859169] SELinux: 2048 avtab hash slots, 112735 rules.
[   68.882094] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   68.882098] SELinux:  130 classes, 112735 rules
[   68.885814] SELinux:  Converting 1337 SID table entries...
[   93.755199] SELinux: 2048 avtab hash slots, 112795 rules.
[   93.791316] SELinux: 2048 avtab hash slots, 112795 rules.
[   93.809913] SELinux:  8 users, 14 roles, 5047 types, 316 bools, 1 sens, 1024 cats
[   93.809916] SELinux:  130 classes, 112795 rules
[   93.812993] SELinux:  Converting 1338 SID table entries...
[   99.696803] type=1305 audit(1678908324.575:188): audit_pid=0 old=751 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   99.703269] type=1130 audit(1678908324.583:189): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   99.703284] type=1131 audit(1678908324.583:190): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   99.814557] type=1305 audit(1678908324.694:191): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   99.814567] type=1305 audit(1678908324.694:192): audit_pid=9818 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 1479.203755] .NET Long Runni[24512]: segfault at 0 ip 00007f6c21d3f31f sp 00007f2b75feb410 error 6 in gssntlmssp.so[7f6c21d39000+1c000]
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
Looking for files matching core.* ...
+ export _commandExitCode=139
+ _commandExitCode=139
+ /bin/python3 /mnt/work/AFB8097B/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 3832409 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhNWY3YjU2ZS1mY2M5LTQzYWQtODYwOC04N2M0NDcwOTIwOTMiLCJzaWQiOiI0NmY2NTNkOC03MDBlLTQxMTItYTA5Ny04YjVmNDMyYTQ1OWUiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzIwNTkxNiIsImpvYnJlZiI6IjQ2OTM5ZDM3LTBiMDYtNGI3YS1iYTg1LWM1ZDcxMWJkYTIyOTo3MWFmZTc2OC1lNDkzLTU5N2MtMjI4Yy0wMzc4MzQxOTNhZWIiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMjA1OTE2Iiwib3JjaGlkIjoiNDY5MzlkMzctMGIwNi00YjdhLWJhODUtYzVkNzExYmRhMjI5LmxpYnJhcmllc190ZXN0X3J1bl9yZWxlYXNlX2NvcmVjbHJfbGludXhfeDY0X2RlYnVnLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjc4OTA3MzE0LCJleHAiOjE2Nzg5MTc1MTR9.Jw6444X6tGPly4azNQx08VtK4UBqtdp12wvNQix0XvW8k976BHgPX9pz3oj2GGV1zjDx8HGHMbUr-358uL1q7gGGdIdlzujkUoKaEttl2aCtdbWr6fiDdHR4aQWDgaIacp0Tb1AR61EG7NetmSlQoYPu-az6572Tjbsi17ULmbYm3Fxm_eWmhJ89ZO20jDNKu2ukWXJYPRjM3LRNgQ3bcucNQBkrZgKexmdmbBQ0MHsNv2b4RzzRQfBoZtyANz3JuExaC4Bh-ueBLei90NhYJYi9UBKpMYemRBJ9lr-OXtsWZBtK416V5IxmfHHu4C4WGRDDp6eBBC35aEGzsB9K5w
2023-03-15T19:48:34.809Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-03-15T19:48:34.809Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/AFB8097B/w/AEB109CE/e' for test results files
2023-03-15T19:48:34.809Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/AFB8097B/w/AEB109CE/uploads' for test results files
2023-03-15T19:48:34.809Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-03-15T19:48:34.810Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/mnt/work/AFB8097B/w/AEB109CE/e/__test_report.json'
2023-03-15T19:48:34.810Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1572 bytes
+ /bin/python3 /mnt/work/AFB8097B/p/gen-debug-dump-docs.py -buildid 205916 -workitem System.Net.Security.Unit.Tests -jobid e1438f85-bd3c-4620-ae7b-ca686d95a426 -outdir /mnt/work/AFB8097B/w/AEB109CE/uploads -templatedir /mnt/work/AFB8097B/p -dumpdir /home/helixbot/dotnetbuild/dumps -productver 7.0.5
read file: /mnt/work/AFB8097B/p/debug-dump-template.md
writing output file: /mnt/work/AFB8097B/w/AEB109CE/uploads/how-to-debug-dump.md
done writing debug dump information
+ exit 139
['System.Net.Security.Unit.Tests' END OF WORK ITEM LOG: Command exited with 139]

Metadata

Metadata

Assignees

No one assigned

    Labels

    arch-x64area-System.Netarea-VM-coreclros-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtime

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions