Skip to content

System.Net native crash in mono linux x64 #83481

@carlossanlop

Description

@carlossanlop

Found in a seemingly unrelated dependency flow PR for 7.0: #83467
Edit: Found also in main: #83356

There is also a crash in these tests, but affecting coreclr, and also happens in both main and 7.0: #83482

It's affecting two sets of System.Net tests:

It's affecting two sets of System.Net tests:

  • System.Net.Mail.Functional: log file
  • System.Net.Security.Unit.Tests: log file <- This callstack seems to be more complete.
Callstack of System.Net.Mail.Functional (expand)
Console log: 'System.Net.Mail.Functional.Tests' from job 0151369d-439e-46a3-8c46-14954ebd0997 workitem 96f87dfb-7846-4bc9-a867-96cf4fd64085 (redhat.7.amd64.open.svc) executed on machine a0009LT running Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo
+ ./RunTests.sh --runtime-path /mnt/work/997C0895/p
----- start Wed Mar 15 19:42:50 UTC 2023 =============== To repro directly: =====================================================
pushd .
/mnt/work/997C0895/p/dotnet exec --runtimeconfig System.Net.Mail.Functional.Tests.runtimeconfig.json --depsfile System.Net.Mail.Functional.Tests.deps.json xunit.console.dll System.Net.Mail.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/mnt/work/997C0895/w/B1400973/e /mnt/work/997C0895/w/B1400973/e
  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 153 of 156 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f45c9ae4182 - Unknown
	0x7f45c9a8a88e - Unknown
	0x7f45c99ea401 - Unknown
	0x7f45ca9a9630 - Unknown
	0x7f45ae9c431f - Unknown
	0x7f45ae9c5b2a - Unknown
	0x7f45ae9cc394 - Unknown
	0x7f45afdc8ecb - Unknown
	0x7f45afdef6ef - Unknown
	0x7f45afdf146a - Unknown
	0x7f45afdc8ecb - Unknown
	0x7f45b96cccda - Unknown
	0x7f45b96cc880 - Unknown
	0x40459c94 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
Missing separate debuginfo for /mnt/work/997C0895/p/dotnet
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/a4/afbd00c00f3229ad5c110a6c0c2dec61dccb65.debug
[New LWP 23618]
[New LWP 23617]
[New LWP 23612]
[New LWP 23611]
[New LWP 23610]
[New LWP 23609]
[New LWP 23608]
[New LWP 23607]
[New LWP 23602]
[New LWP 23601]
[New LWP 23600]
[New LWP 23599]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /mnt/work/997C0895/p/host/fxr/7.0.5/libhostfxr.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/14/24382a6586a7e3eac1a3afb5d615f4e16f476b.debug
Missing separate debuginfo for /mnt/work/997C0895/p/shared/Microsoft.NETCore.App/7.0.5/libhostpolicy.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/1c/c50e099f7b9ae16c6e725d0dbdf3ac697cae29.debug
Missing separate debuginfo for /lib64/libwbclient.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/bd/53a175ff68e6a67f5c2237b15d0b3a9b03f665.debug
0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  Id   Target Id         Frame 
  13   Thread 0x7f45c2bff700 (LWP 23599) "SGen worker" 0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12   Thread 0x7f45c0e0f700 (LWP 23600) "dotnet" 0x00007f45c9c9bddd in poll () from /lib64/libc.so.6
  11   Thread 0x7f45c0c0e700 (LWP 23601) "Finalizer" 0x00007f45ca9a7b3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  10   Thread 0x7f45b9e40700 (LWP 23602) "dotnet" 0x00007f45ca9a875d in read () from /lib64/libpthread.so.0
  9    Thread 0x7f45b9da6700 (LWP 23607) ".NET Long Runni" 0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8    Thread 0x7f45b9ba5700 (LWP 23608) ".NET ThreadPool" 0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7    Thread 0x7f45ad14f700 (LWP 23609) ".NET ThreadPool" 0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6    Thread 0x7f45b8894700 (LWP 23610) ".NET ThreadPool" 0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7f45b9095700 (LWP 23611) ".NET Long Runni" 0x00007f45ca9a91d9 in waitpid () from /lib64/libpthread.so.0
  4    Thread 0x7f45ad0fe700 (LWP 23612) ".NET Long Runni" 0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7f45acc7b700 (LWP 23617) ".NET Sockets" 0x00007f45c9ca70e3 in epoll_wait () from /lib64/libc.so.6
  2    Thread 0x7f45aca7a700 (LWP 23618) ".NET Timer" 0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1    Thread 0x7f45cadc7780 (LWP 23598) "dotnet" 0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thread 13 (Thread 0x7f45c2bff700 (LWP 23599)):
#0  0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c99d1af3 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#2  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#3  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#4  0x00007f45ca9a1ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f45c9ca6b0d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f45c0e0f700 (LWP 23600)):
#0  0x00007f45c9c9bddd in poll () from /lib64/libc.so.6
#1  0x00007f45c9b7d6fa in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0x7f45bc002250, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7f45c9b7cb30 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x00007f45c9b7ac05 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7f45c9b7cb30 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x00007f45c9b79319 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x00007f45c9b7cb11 in ep_rt_thread_mono_start_func (data=0x563e9adce240) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x00007f45ca9a1ea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f45c9ca6b0d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f45c0c0e700 (LWP 23601)):
#0  0x00007f45ca9a7b3b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007f45ca9a7bcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f45ca9a7c6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f45c9952206 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x00007f45c992b23a in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x00007f45c992b0c9 in start_wrapper (data=0x563e9ade0120) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x00007f45ca9a1ea5 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f45c9ca6b0d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f45b9e40700 (LWP 23602)):
#0  0x00007f45ca9a875d in read () from /lib64/libpthread.so.0
#1  0x00007f45c07e470e in SignalHandlerLoop (arg=0x563e9b5f2fb0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x00007f45ca9a1ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f45c9ca6b0d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f45b9da6700 (LWP 23607)):
#0  0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e52fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7f45b0174dc0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#2  0x000000004029032c in ?? ()
#3  0x00007f45c2d8eef8 in ?? ()
#4  0xffffffffffffffff in ?? ()
#5  0x00007f45ad29cfb0 in ?? ()
#6  0x0000000000000001 in ?? ()
#7  0x00007f45c2d8ef28 in ?? ()
#8  0xffffffffffffffff in ?? ()
#9  0x00007f45ad29cfb0 in ?? ()
#10 0x00007f45b0174620 in ?? ()
#11 0x00007f45b9da58b0 in ?? ()
#12 0x00007f45b9da57c0 in ?? ()
#13 0x00007f45b00008c0 in ?? ()
#14 0x000000004029015c in ?? ()
#15 0x00007f45b9da5930 in ?? ()
#16 0x000000004029011c in ?? ()
#17 0x00007f45c2d8ef28 in ?? ()
#18 0x000000004028f7a8 in ?? ()
#19 0x00007f45b9da5930 in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f45b9ba5700 (LWP 23608)):
#0  0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e5440 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f45a8038f10, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#2  0x00000000402f4357 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f45ad14f700 (LWP 23609)):
#0  0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e5440 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f45a40052e0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#2  0x00000000402f4357 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f45b8894700 (LWP 23610)):
#0  0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c997d594 in mono_os_cond_timedwait (cond=0x7f45b8893980, mutex=0x7f45b0183df0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#2  0x00007f45c9982d44 in mono_coop_cond_timedwait (cond=0x7f45b8893980, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#3  mono_lifo_semaphore_timed_wait (semaphore=0x7f45b0183df0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#4  0x00000000402f70f7 in ?? ()
#5  0x0000000000000002 in ?? ()
#6  0x0000000000000046 in ?? ()
#7  0x00007f45c2d928b8 in ?? ()
#8  0x00007f45c2d928b8 in ?? ()
#9  0x0000000000004e20 in ?? ()
#10 0x00007f459c000fd0 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007f45b88939f0 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f45b9095700 (LWP 23611)):
#0  0x00007f45ca9a91d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f45c9ae42c7 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7f45b908f9b0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x00007f45c9a8a88e in mono_handle_native_crash (signal=0x7f45c9809dbf "SIGSEGV", mctx=0x7f45b908f9b0, info=0x7f45b908fc70) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x00007f45c99ea401 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7f45b908fc70, context=0x7f45b908fb40, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3773
#5  <signal handler called>
#6  0x00007f45ae9c431f in ntlm_decode_u16l_str_hdr.isra.7 () from /usr/lib64/gssntlmssp/gssntlmssp.so
#7  0x00007f45ae9c5b2a in ntlm_decode_chal_msg () from /usr/lib64/gssntlmssp/gssntlmssp.so
#8  0x00007f45ae9cc394 in gssntlm_init_sec_context () from /usr/lib64/gssntlmssp/gssntlmssp.so
#9  0x00007f45afdc8ecb in gss_init_sec_context () from /lib64/libgssapi_krb5.so.2
#10 0x00007f45afdef6ef in init_ctx_call_init () from /lib64/libgssapi_krb5.so.2
#11 0x00007f45afdf146a in spnego_gss_init_sec_context () from /lib64/libgssapi_krb5.so.2
#12 0x00007f45afdc8ecb in gss_init_sec_context () from /lib64/libgssapi_krb5.so.2
#13 0x00007f45b96cccda in NetSecurityNative_InitSecContextEx (minorStatus=0x7f45b9090d30, claimantCredHandle=0x7f45a030a440, contextHandle=0x7f45b9090b18, packageType=0, cbt=0x0, cbtSize=0, targetName=0x7f45a03346b0, reqFlags=48, inputBytes=0x7f45c2cadc00 "\241u0s\240\003\n\001\003\241\f\006\n+\006\001\004\001\202\067\002\002\n\242^\004\\NTLMSSP", inputLength=119, outBuffer=0x7f45b9090d40, retFlags=0x7f45b9090d00, isNtlmUsed=0x7f45b9090b10) at /__w/1/s/src/native/libs/System.Net.Security.Native/pal_gssapi.c:374
#14 0x00007f45b96cc880 in NetSecurityNative_InitSecContext (minorStatus=0x7f45b9090d30, claimantCredHandle=0x7f45a030a440, contextHandle=0x7f45b9090b18, packageType=0, targetName=0x7f45a03346b0, reqFlags=48, inputBytes=0x7f45c2cadc00 "\241u0s\240\003\n\001\003\241\f\006\n+\006\001\004\001\202\067\002\002\n\242^\004\\NTLMSSP", inputLength=119, outBuffer=0x7f45b9090d40, retFlags=0x7f45b9090d00, isNtlmUsed=0x7f45b9090b10) at /__w/1/s/src/native/libs/System.Net.Security.Native/pal_gssapi.c:288
#15 0x0000000040459c94 in ?? ()
#16 0x00007f45c2cadc00 in ?? ()
#17 0x0000000000000077 in ?? ()
#18 0x00007f45b9090d40 in ?? ()
#19 0x00007f45b9090d00 in ?? ()
#20 0x00007f45b9090b10 in ?? ()
#21 0x00007f45b9090980 in ?? ()
#22 0x00007f45a00008c0 in ?? ()
#23 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f45ad0fe700 (LWP 23612)):
#0  0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e5440 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f4594002420, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#2  0x00000000402f4357 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f45acc7b700 (LWP 23617)):
#0  0x00007f45c9ca70e3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f45c07e000e in WaitForSocketEventsInner (port=9, buffer=0x7f45a0252cd0, count=0x7f45acc7abf8) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2723
#2  0x00007f45c07dff2f in SystemNative_WaitForSocketEvents (port=9, buffer=0x7f45a0252cd0, count=0x7f45acc7abf8) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3023
#3  0x0000000040419dc9 in ?? ()
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f45aca7a700 (LWP 23618)):
#0  0x00007f45ca9a5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e5440 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f458c005360, timeoutMilliseconds=99976) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#2  0x00000000402f4357 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f45cadc7780 (LWP 23598)):
#0  0x00007f45ca9a5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f45c07e52fc in SystemNative_LowLevelMonitor_Wait (monitor=0x563e9b85d000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#2  0x000000004029032c in ?? ()
#3  0x00007f45ad2cffb0 in ?? ()
#4  0xffffffffffffffff in ?? ()
#5  0x00007f45c2d9b280 in ?? ()
#6  0x0000000000000001 in ?? ()
#7  0x00007f45ad2cffe0 in ?? ()
#8  0xffffffffffffffff in ?? ()
#9  0x00007f45c2d9b280 in ?? ()
#10 0x0000563e9adc5840 in ?? ()
#11 0x00007ffd5c05e6b0 in ?? ()
#12 0x00007ffd5c05e5c0 in ?? ()
#13 0x0000563e9ad177e0 in ?? ()
#14 0x000000004029015c in ?? ()
#15 0x00007ffd5c05e730 in ?? ()
#16 0x000000004029011c in ?? ()
#17 0x00007f45ad2cffe0 in ?? ()
#18 0x000000004028f7a8 in ?? ()
#19 0x00007ffd5c05e730 in ?? ()
#20 0x0000000000000000 in ?? ()
[Inferior 1 (process 23598) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f45ae9c431f):0x7f45ae9c430f  41 5e 41 5f c3 0f 1f 40 00 31 c0 45 31 e4 31 db  A^[email protected].
0x7f45ae9c431f  c6 00 00 4c 89 65 00 89 d8 eb c7 66 0f 1f 44 00  ...L.e.....f..D.
0x7f45ae9c432f  00 48 8b 4a 08 49 39 cf 77 b8 41 89 f0 49 89 fd  .H.J.I9.w.A..I..
0x7f45ae9c433f  0f b7 fb 41 f7 d0 41 39 f8 72 a7 01 fe 48 39 f1  ...A..A9.r...H9.

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at NetSecurityNative:<InitSecContext>g____PInvoke|16_0 <0x00103>
	  at NetSecurityNative:InitSecContext <0x002b7>
	  at NetSecurityNative:InitSecContext <0x0009f>
	  at System.Net.Security.NegotiateStreamPal:EstablishSecurityContext <0x004a3>
	  at System.Net.Security.NegotiateStreamPal:InitializeSecurityContext <0x00107>
	  at System.Net.NTAuthentication:GetOutgoingBlob <0x0021f>
	  at System.Net.Security.NegotiateAuthentication:GetOutgoingBlob <0x00097>
	  at System.Net.Security.NegotiateAuthentication:GetOutgoingBlob <0x0009b>
	  at System.Net.Mail.SmtpNegotiateAuthenticationModule:Authenticate <0x002db>
	  at System.Net.Mail.SmtpConnection:GetConnection <0x008f7>
	  at System.Net.Mail.SmtpTransport:GetConnection <0x0025b>
	  at System.Net.Mail.SmtpClient:GetConnection <0x00063>
	  at System.Net.Mail.SmtpClient:Send <0x0092b>
	  at System.Net.Mail.Tests.SmtpClientTest:TestGssapiAuthentication <0x00157>
	  at System.Object:runtime_invoke_void__this__ <0x00091>
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x000b9>
	  at System.Reflection.MethodInvoker:InterpretedInvoke <0x000b7>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x00253>
	  at System.Reflection.MethodBase:Invoke <0x00049>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00043>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x003b3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000b7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0002b>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x000c7>
	  at <AggregateAsync>d__4:MoveNext <0x000e2>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000bb>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000bf>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00157>
	  at <RunAsync>d__9:MoveNext <0x00095>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000bb>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00297>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000c3>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x000ef>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000f3>
	  at <<RunAsync>b__47_0>d:MoveNext <0x006b6>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000cb>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x000cf>
	  at <RunAsync>d__10`1:MoveNext <0x0009c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000bb>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x000f3>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000db>
	  at <InvokeTestAsync>d__4:MoveNext <0x001ed>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000c3>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000c7>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00038>
	  at <RunAsync>d__10`1:MoveNext <0x000c3>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000b7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000f3>
	  at <RunAsync>d__43:MoveNext <0x0050f>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000cb>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x000cf>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x000d7>
	  at <RunAsync>d__19:MoveNext <0x00393>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000c3>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x000cb>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x000b3>
	  at Microsoft.DotNet.XUnitExtensions.SkippedFactTestCase:<>n__0 <0x0005b>
	  at <RunAsync>d__2:MoveNext <0x00123>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000df>
	  at Microsoft.DotNet.XUnitExtensions.SkippedFactTestCase:RunAsync <0x00167>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x000a5>
	  at <RunTestCasesAsync>d__32:MoveNext <0x001bb>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000cb>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x000cf>
	  at <RunAsync>d__31:MoveNext <0x001d9>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000b7>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x000c7>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000eb>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x008a5>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000d3>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x000d7>
	  at <RunAsync>d__37:MoveNext <0x003c2>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000c3>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x000cb>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000f7>
	  at <RunTestClassesAsync>d__28:MoveNext <0x003a1>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000cb>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x000cf>
	  at <RunAsync>d__27:MoveNext <0x003bf>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000c3>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x000cb>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x000c7>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00060>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00078>
	  at <>c:<.cctor>b__273_0 <0x0002f>
	  at System.Threading.ExecutionContext:RunInternal <0x000e2>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x002b3>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000d7>
	  at <>c:<.cctor>b__8_0 <0x00057>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0004f>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x0004b>
	  at System.Threading.ExecutionContext:RunInternal <0x000e2>
	  at System.Threading.ExecutionContext:Run <0x0004b>
	  at System.Object:lambda_method2 <0x0008a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x0005e>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x001f7>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x0006b>
	  at System.Threading.Tasks.Task:InnerInvoke <0x000a2>
	  at <>c:<.cctor>b__273_0 <0x0002f>
	  at System.Threading.ExecutionContext:RunInternal <0x000e2>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x002b3>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000a3>
	  at <>c:<.cctor>b__10_0 <0x0005b>
	  at System.Threading.Thread:StartCallback <0x0012e>
	  at System.Object:runtime_invoke_void__this__ <0x00091>
=================================================================
./RunTests.sh: line 168: 23598 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Mail.Functional.Tests.runtimeconfig.json --depsfile System.Net.Mail.Functional.Tests.deps.json xunit.console.dll System.Net.Mail.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/mnt/work/997C0895/w/B1400973/e
----- end Wed Mar 15 19:42:53 UTC 2023 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
[    4.272553] cryptd: max_cpu_qlen set to 1000
[    4.356475] AVX2 version of gcm_enc/dec engaged.
[    4.356475] AES CTR mode by8 optimization enabled
[    4.398991] XFS (dm-4): Mounting V5 Filesystem
[    4.401508] XFS (dm-0): Mounting V5 Filesystem
[    4.415658] XFS (sda2): Mounting V5 Filesystem
[    4.428185] XFS (dm-2): Mounting V5 Filesystem
[    4.433176] XFS (sda2): Ending clean mount
[    4.444656] XFS (dm-3): Mounting V5 Filesystem
[    4.450033] psmouse serio1: trackpoint: IBM TrackPoint firmware: 0x01, buttons: 3/3
[    4.450294] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    4.451304] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input4
[    4.452308] input: AT Translated Set 2 keyboard as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:08/VMBUS:01/d34b2567-b9b6-42b9-8778-0a4ec0b955bf/serio2/input/input5
[    4.458248] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
[    4.467050] XFS (dm-3): Ending clean mount
[    4.567963] XFS (dm-0): Ending clean mount
[    4.583107] XFS (dm-2): Ending clean mount
[    4.810152] XFS (dm-4): Ending clean mount
[    4.829035] systemd-journald[538]: Received request to flush runtime journal from PID 1
[    4.901073] type=1305 audit(1678908748.920:4): audit_pid=749 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[    6.281520] UDF-fs: INFO Mounting volume 'UDF Volume', timestamp 2023/03/16 00:00 (1000)
[    6.408474] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.408482] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.074566] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    7.505637] hv_utils: KVP IC version 4.0
[    8.522388]  sdb: sdb1
[    9.354405] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[   12.945545] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   51.250024] hv_balloon: Max. dynamic memory size: 8192 MB
[   54.116020] SELinux: 2048 avtab hash slots, 112733 rules.
[   54.159225] SELinux: 2048 avtab hash slots, 112733 rules.
[   54.183604] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   54.183607] SELinux:  130 classes, 112733 rules
[   54.186834] SELinux:  Converting 1337 SID table entries...
[   61.360549] SELinux: 2048 avtab hash slots, 112735 rules.
[   61.395923] SELinux: 2048 avtab hash slots, 112735 rules.
[   61.419639] SELinux:  8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
[   61.419643] SELinux:  130 classes, 112735 rules
[   61.424119] SELinux:  Converting 1337 SID table entries...
[   86.911476] SELinux: 2048 avtab hash slots, 112795 rules.
[   86.949884] SELinux: 2048 avtab hash slots, 112795 rules.
[   86.974845] SELinux:  8 users, 14 roles, 5047 types, 316 bools, 1 sens, 1024 cats
[   86.974848] SELinux:  130 classes, 112795 rules
[   86.979424] SELinux:  Converting 1338 SID table entries...
[   95.814033] type=1305 audit(1678908839.832:183): audit_pid=0 old=749 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   95.820203] type=1130 audit(1678908839.839:184): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   95.820217] type=1131 audit(1678908839.839:185): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   96.146062] type=1305 audit(1678908840.165:186): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[   96.146074] type=1305 audit(1678908840.165:187): audit_pid=9804 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[  328.277528] .NET Long Runni[17415]: segfault at 0 ip 00007fc1b59da31f sp 00007f810b4d0aa0 error 6 in gssntlmssp.so[7fc1b59d4000+1c000]
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
Looking for files matching core.* ...
+ export _commandExitCode=134
+ _commandExitCode=134
+ /bin/python3 /mnt/work/997C0895/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 3832400 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJmYTNiMmI3Ny04OWE1LTQzMWQtYTU1NS1mZDg2Y2ViMWI3YTIiLCJzaWQiOiI4OWVlMDkzZS0wNGZhLTQzMTYtOTJmZi00MTY4MGVmODNjMDEiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzIwNTkxNiIsImpvYnJlZiI6IjQ2OTM5ZDM3LTBiMDYtNGI3YS1iYTg1LWM1ZDcxMWJkYTIyOTpiNDM0NGIwZC0wZjkyLTVkNjktY2NhZi1lMGIyNGZiZjE0YTIiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMjA1OTE2Iiwib3JjaGlkIjoiNDY5MzlkMzctMGIwNi00YjdhLWJhODUtYzVkNzExYmRhMjI5LmxpYnJhcmllc190ZXN0X3J1bl9yZWxlYXNlX21vbm9fbGludXhfeDY0X2RlYnVnLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjc4OTA3MzA1LCJleHAiOjE2Nzg5MTc1MDV9.tbR-OShUXcmXRJoH3UvFIs5x_94srTxN2h0RhdvIUoNDsOhZjuvdGtK14SGxKw17VQNeW5QGIuyE30ec-U6pcP7wINQdk9vURegaz0RL3v0XLiaUpO3eVS78FMMRGmY4IWvZTpUfolyZnNAJtO1y7AmqbwcGyu8SpGGmZ0OIb4BjDy_PO_Dv2h7KFPuide9mVNSR0d7GO2KaSd6M0nPk-iXod2rSy5OVmJ0QePpH4dgEAScqOshk-_AwxQXV7xAoSEu_z1UzO8vZO8Wobg3jObCLDC5R2o-EL7y6t5mkmO1u09pVqjdz9OFKyN8HDo6Cyy90WaWoFdR5TdSKx9BgDA
2023-03-15T19:43:04.326Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-03-15T19:43:04.326Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/997C0895/w/B1400973/e' for test results files
2023-03-15T19:43:04.327Z	INFO   	run.py	__init__(42)	read_results	Searching '/mnt/work/997C0895/w/B1400973/uploads' for test results files
2023-03-15T19:43:04.327Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-03-15T19:43:04.327Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/mnt/work/997C0895/w/B1400973/e/__test_report.json'
2023-03-15T19:43:04.327Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1568 bytes
+ exit 134
['System.Net.Mail.Functional.Tests' END OF WORK ITEM LOG: Command exited with 134]

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions