Skip to content

Conversation

bartonjs
Copy link
Member

@bartonjs bartonjs commented Jul 3, 2025

EVP_aes_###_wrap_pad is available starting from 1.0.1, so here we treat it as a Required import.

Android's copy of AesImplementation is split off (as a copy) since it doesn't really use OpenSSL, just piggy-backed off of that PAL.

Fixes #108332

@bartonjs bartonjs added this to the 10.0.0 milestone Jul 3, 2025
@bartonjs bartonjs self-assigned this Jul 3, 2025
@Copilot Copilot AI review requested due to automatic review settings July 3, 2025 01:45
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for the OpenSSL AES Key Wrap with Padding (EVP_aes_##_wrap_pad) on Linux/Unix and introduces a corresponding managed implementation along with an Android-specific AES class split from the OpenSSL one.

  • Native shims and exports for EVP_aes_*_wrap_pad were added across the PAL headers, sources, shim, and entrypoints.
  • Managed wrappers (EncryptKeyWrapPaddedCore, DecryptKeyWrapPaddedCore, and KeyWrap) were implemented in AesImplementation.OpenSsl.cs, and an Android-specific AES implementation file was introduced.
  • Project and interop files were updated to include the new native and managed methods.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
pal_evp_cipher.h, pal_evp_cipher.c Added declarations and definitions for CryptoNative_EvpAes*WrapPad
opensslshim.h Registered EVP_aes_*_wrap_pad as a required OpenSSL symbol
entrypoints.c Exported the new native wrap-pad functions for P/Invoke
AesImplementation.OpenSsl.cs Added managed AES key-wrap-pad methods and key-wrap algorithm lookup
AesImplementation.Android.cs Introduced Android‐specific AES implementation split
System.Security.Cryptography.csproj Replaced OpenSSL AES implementation include with Android one
Interop.EVP.Cipher.cs Imported new native wrap-pad entrypoints for Unix interop
Comments suppressed due to low confidence (2)

src/libraries/System.Security.Cryptography/src/System.Security.Cryptography.csproj:1223

  • The csproj entry for AesImplementation.OpenSsl.cs was removed and replaced with only the Android implementation, causing the OpenSSL-based AES code to no longer be compiled. You should include both AesImplementation.OpenSsl.cs and AesImplementation.Android.cs, or add a condition so that each file is compiled on the appropriate platform.
             Link="Common\System\Security\Cryptography\SP800108HmacCounterKdfImplementationManaged.cs" />

src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/AesImplementation.OpenSsl.cs:42

  • There are no unit tests covering the new EncryptKeyWrapPaddedCore and DecryptKeyWrapPaddedCore methods. Consider adding tests in the existing AES or OpenSSL test suite to verify key-wrap padding behavior for 128, 192, and 256-bit keys.
        protected override void EncryptKeyWrapPaddedCore(ReadOnlySpan<byte> source, Span<byte> destination)

Copy link
Member

@vcsjones vcsjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really just one question.

@vcsjones
Copy link
Member

vcsjones commented Jul 3, 2025

I also want to point out the EVP_CIPHER_CTX_FLAG_WRAP_ALLOW flag. It seems unused in OpenSSL 3, but older OpenSSLs might need that flag set on the cipher context. Docs

EVP_CIPHER_CTX_FLAG_WRAP_ALLOW flag treated specially in EVP_CipherInit_ex().

It seems that 1.1.1 will check for this and not allow wrapping. https://github.com/openssl/openssl/blob/b372b1f76450acdfed1e2301a39810146e28b02c/crypto/evp/evp_enc.c#L160-L164

@bartonjs
Copy link
Member Author

bartonjs commented Jul 3, 2025

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

This also adds error checking when creating the context.
@vcsjones
Copy link
Member

vcsjones commented Jul 4, 2025

The native crashes on OpenSSL 1.1 are an interesting confluence of things.

  1. OpenSSL 1.1 requires you to specify EVP_CIPHER_CTX_FLAG_WRAP_ALLOW on the context before you call EVP_CipherInit_ex. Otherwise you get an error.
  2. We were getting that error because we weren't specifying that flag.
  3. We were returning a NULL context.
  4. The managed side did not check if that context handle it got back was valid or not. It dutifully passed the null handle to EVP_CipherUpdate.
  5. Null dereference from OpenSSL.

Here is a pull request to your branch with my suggested changes. bartonjs#10

@bartonjs
Copy link
Member Author

bartonjs commented Jul 7, 2025

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@bartonjs
Copy link
Member Author

  • None of the reported SIGSEGV show any of the new code being involved.
    • At least, as far as I can read them.
  • All failures involved Mono, they passed if CoreCLR was used.
  • Failure on net10.0-linux-Debug-arm64-Mono_Minijit_Debug-(AzureLinux.3.0.ArmArch.Open)[email protected]/dotnet-buildtools/prereqs:azurelinux-3.0-helix-arm64v8 leads one to suspect "hmm, AZ3/SCOSSL problem?"
    • But it passed on net10.0-linux-Debug-x64-coreclr_release-(AzureLinux.3.0.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

I'll see if pulling in a newer main build happens to have a mono fix that makes the problem go away. Otherwise, probably need someone familiar with mono to take a look.

@bartonjs
Copy link
Member Author

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@bartonjs
Copy link
Member Author

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@bartonjs
Copy link
Member Author

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@bartonjs
Copy link
Member Author

bartonjs commented Jul 28, 2025

@steveisok Would you happen to have any insight as to the mono-only crashes this PR is seeing? When it crashes, the new code isn't even involved. e.g.

Expand for very big stack

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7295513fd15f - Unknown
	0x7295513a2d6e - Unknown
	0x729551307286 - Unknown
	0x729551842520 - Unknown
	0x7295515c294a - Unknown
	0x7295515c628a - Unknown
	0x7295515e61f8 - Unknown
	0x7295515e51ac - Unknown
	0x729551894ac3 - Unknown
	0x729551926850 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
    System.Security.Cryptography.Tests.FactorySha256Tests.HashData_NotSupported [SKIP]
      Condition(s) not met: "IsNotSupported"
[New LWP 10704]
[New LWP 10705]
[New LWP 10706]
[New LWP 10707]
[New LWP 10713]
[New LWP 10715]
[New LWP 10716]
[New LWP 10717]
[New LWP 10718]
[New LWP 10719]
[New LWP 10836]
[New LWP 11152]
[New LWP 11155]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x60517f24f984) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.
  Id   Target Id                                           Frame 
* 1    Thread 0x729551f08740 (LWP 10703) "dotnet"          __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x60517f24f984) at ./nptl/futex-internal.c:57
  2    Thread 0x7295507ff640 (LWP 10704) "SGen worker"     0x00007295518ea42f in __GI___wait4 (pid=pid@entry=11138, stat_loc=stat_loc@entry=0x7295507fd280, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
  3    Thread 0x72954e3ff640 (LWP 10705) ".NET EventPipe"  0x0000729551918bcf in __GI___poll (fds=fds@entry=0x729548002970, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x72954e1fe640 (LWP 10706) "Finalizer"       __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295516030c0 <finalizer_sem>) at ./nptl/futex-internal.c:57
  5    Thread 0x72954c0dd640 (LWP 10707) ".NET SigHandler" __GI___libc_read (nbytes=1, buf=0x72954c0dce47, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x729544ed9640 (LWP 10713) ".NET Timer"      __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x72952c0083b0) at ./nptl/futex-internal.c:57
  7    Thread 0x72954487b640 (LWP 10715) ".NET TP Gate"    __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295240090f0) at ./nptl/futex-internal.c:57
  8    Thread 0x7295446d2640 (LWP 10716) ".NET Sockets"    0x0000729551925e2e in epoll_wait (epfd=9, events=0x72953004cd80, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  9    Thread 0x7295443c4640 (LWP 10717) ".NET TP Worker"  __futex_abstimed_wait_common64 (private=885317600, cancel=true, abstime=0x7295443c28f8, op=137, expected=0, futex_word=0x72951c099f00) at ./nptl/futex-internal.c:57
  10   Thread 0x7295461ff640 (LWP 10718) ".NET Long Runni" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295343faa50) at ./nptl/futex-internal.c:57
  11   Thread 0x729545ffe640 (LWP 10719) ".NET TP Worker"  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x729545ffbbc8, op=137, expected=0, futex_word=0x729545ffbc48) at ./nptl/futex-internal.c:57
  12   Thread 0x729523868640 (LWP 10836) ".NET TP Worker"  __futex_abstimed_wait_common64 (private=2122004044, cancel=true, abstime=0x729523865588, op=137, expected=0, futex_word=0x7295180704b0) at ./nptl/futex-internal.c:57
  13   Thread 0x729523b6a640 (LWP 11152) ".NET TP Worker"  0x00007295514011af in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523b67fc0, context=context@entry=0x72953000db60, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4501
  14   Thread 0x729523566640 (LWP 11155) ".NET TP Worker"  __futex_abstimed_wait_common64 (private=16, cancel=true, abstime=0x729523564228, op=137, expected=0, futex_word=0x7295235642a8) at ./nptl/futex-internal.c:57

Thread 14 (Thread 0x729523566640 (LWP 11155) ".NET TP Worker"):
#0  __futex_abstimed_wait_common64 (private=16, cancel=true, abstime=0x729523564228, op=137, expected=0, futex_word=0x7295235642a8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=16, abstime=0x729523564228, clockid=0, expected=0, futex_word=0x7295235642a8) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7295235642a8, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x729523564228, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893e9b in __pthread_cond_wait_common (abstime=0x729523564228, clockid=1, mutex=0x729534bc6cd0, cond=0x729523564280) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x729523564280, mutex=mutex@entry=0x729534bc6cd0, abstime=abstime@entry=0x729523564228) at ./nptl/pthread_cond_wait.c:652
#5  0x00007295514b3ad3 in mono_os_cond_timedwait (cond=cond@entry=0x729523564280, mutex=mutex@entry=0x729534bc6cd0, timeout_ms=timeout_ms@entry=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#6  0x00007295514b81a9 in mono_coop_cond_timedwait (cond=0x729523564280, mutex=0x729534bc6cd0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/../../mono/utils/mono-coop-mutex.h:103
#7  mono_lifo_semaphore_timed_wait (semaphore=0x729534bc6cd0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:55
#8  0x000072955141146f in do_icall (sig=0x7295300fb750, op=MINT_ICALLSIG_PP_P, ret_sp=0x729522946270, sp=0x729522946280, ptr=0x72955154c830 <ves_icall_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2451
#9  0x000072955140f9fa in do_icall_wrapper (frame=0x729523565a80, sig=0x7295300fb750, op=MINT_ICALLSIG_PP_P, ret_sp=0x729522946270, sp=0x729522946280, ptr=0x72955154c830 <ves_icall_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal>, save_last_error=0, gc_transitions=0x729523565c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2534
#10 0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523565d20, context=context@entry=0x729510006d00, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#11 0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729523565dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#12 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509c3950, params=0x0, exc=0x0, error=0x729523565dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#13 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509c3950, params=params@entry=0x0, error=error@entry=0x729523565dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#14 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#15 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#16 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#17 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x729523b6a640 (LWP 11152) ".NET TP Worker"):
#0  0x00007295514011af in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523b67fc0, context=context@entry=0x72953000db60, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4501
#1  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729523b68118) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#2  0x0000729551534927 in do_runtime_invoke (method=0x72953457b810, obj=0x729550b42318, params=0x0, exc=0x0, error=0x729523b68118) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#3  mono_runtime_invoke_checked (method=0x72953457b810, obj=0x729550b42318, params=0x0, error=error@entry=0x729523b68118) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#4  0x000072955153d3c1 in mono_runtime_try_invoke_byrefs (method=method@entry=0x72953457b810, obj=0x729523b679f0, obj@entry=0x729550b42318, params_byref=params_byref@entry=0x0, exc=exc@entry=0x0, error=error@entry=0x729523b68118) at /__w/1/s/src/mono/mono/metadata/object.c:5213
#5  0x00007295514f1fcb in ves_icall_InternalInvoke (method_handle=method_handle@entry=..., this_arg_handle=this_arg_handle@entry=..., params_byref=params_byref@entry=0x0, exception_out=exception_out@entry=..., error=error@entry=0x729523b68118) at /__w/1/s/src/mono/mono/metadata/icall.c:3667
#6  0x00007295514fc24b in ves_icall_InternalInvoke_raw (a0=0x729522847470, a1=0x729522847478, a2=0x0, a3=0x729522847490) at /__w/1/s/src/mono/mono/metadata/icall-def.h:369
#7  0x00007295514113fb in do_icall (sig=0x729534201378, op=MINT_ICALLSIG_PPPP_P, ret_sp=0x7295228474a0, sp=0x7295228474b0, ptr=0x7295514fc1f0 <ves_icall_InternalInvoke_raw>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2475
#8  0x000072955140f9fa in do_icall_wrapper (frame=0x729523b68a30, sig=0x729534201378, op=MINT_ICALLSIG_PPPP_P, ret_sp=0x7295228474a0, sp=0x7295228474b0, ptr=0x7295514fc1f0 <ves_icall_InternalInvoke_raw>, save_last_error=0, gc_transitions=0x729523b69c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2534
#9  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523b69d20, context=context@entry=0x72953000db60, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#10 0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729523b69dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#11 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x729550995b20, params=0x0, exc=0x0, error=0x729523b69dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#12 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x729550995b20, params=params@entry=0x0, error=error@entry=0x729523b69dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#13 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#14 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#15 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 12 (Thread 0x729523868640 (LWP 10836) ".NET TP Worker"):
#0  __futex_abstimed_wait_common64 (private=2122004044, cancel=true, abstime=0x729523865588, op=137, expected=0, futex_word=0x7295180704b0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=2122004044, abstime=0x729523865588, clockid=1077128362, expected=0, futex_word=0x7295180704b0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7295180704b0, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x729523865588, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893e9b in __pthread_cond_wait_common (abstime=0x729523865588, clockid=1, mutex=0x729518070460, cond=0x729518070488) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x729518070488, mutex=0x729518070460, abstime=0x729523865588) at ./nptl/pthread_cond_wait.c:652
#5  0x00007295511cb24d in SystemNative_LowLevelMonitor_TimedWait (monitor=0x729518070460, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:193
#6  0x000072955141146f in do_icall (sig=0x7295349f1dd8, op=MINT_ICALLSIG_PP_P, ret_sp=0x72952386ba30, sp=0x72952386ba40, ptr=0x7295511cb170 <SystemNative_LowLevelMonitor_TimedWait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2451
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x729523865930, sig=0x7295349f1dd8, op=MINT_ICALLSIG_PP_P, ret_sp=0x72952386ba30, sp=0x72952386ba40, ptr=0x7295511cb170 <SystemNative_LowLevelMonitor_TimedWait>, save_last_error=0, gc_transitions=0x729523865c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523865d20, context=context@entry=0x729518004230, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729523865e78) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x729534612d50, obj=0x729550983760, params=0x0, exc=0x0, error=0x729523865e78) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=0x729534612d50, obj=0x729550983760, params=0x0, error=error@entry=0x729523865e78) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955153d3c1 in mono_runtime_try_invoke_byrefs (method=method@entry=0x729534612d50, obj=0x89, obj@entry=0x729550983760, params_byref=params_byref@entry=0x0, exc=exc@entry=0x0, error=error@entry=0x729523865e78) at /__w/1/s/src/mono/mono/metadata/object.c:5213
#13 0x00007295514f1fcb in ves_icall_InternalInvoke (method_handle=method_handle@entry=..., this_arg_handle=this_arg_handle@entry=..., params_byref=params_byref@entry=0x0, exception_out=exception_out@entry=..., error=error@entry=0x729523865e78) at /__w/1/s/src/mono/mono/metadata/icall.c:3667
#14 0x00007295514fc24b in ves_icall_InternalInvoke_raw (a0=0x72952386b3b0, a1=0x72952386b3b8, a2=0x0, a3=0x72952386b3d0) at /__w/1/s/src/mono/mono/metadata/icall-def.h:369
#15 0x00007295514113fb in do_icall (sig=0x729534201378, op=MINT_ICALLSIG_PPPP_P, ret_sp=0x72952386b3e0, sp=0x72952386b3f0, ptr=0x7295514fc1f0 <ves_icall_InternalInvoke_raw>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2475
#16 0x000072955140f9fa in do_icall_wrapper (frame=0x7295238668b0, sig=0x729534201378, op=MINT_ICALLSIG_PPPP_P, ret_sp=0x72952386b3e0, sp=0x72952386b3f0, ptr=0x7295514fc1f0 <ves_icall_InternalInvoke_raw>, save_last_error=0, gc_transitions=0x729523867c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2534
#17 0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729523867d20, context=context@entry=0x729518004230, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#18 0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729523867dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#19 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x72955098f3d8, params=0x0, exc=0x0, error=0x729523867dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#20 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x72955098f3d8, params=params@entry=0x0, error=error@entry=0x729523867dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#21 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#22 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#23 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#24 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x729545ffe640 (LWP 10719) ".NET TP Worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x729545ffbbc8, op=137, expected=0, futex_word=0x729545ffbc48) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x729545ffbbc8, clockid=0, expected=0, futex_word=0x729545ffbc48) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x729545ffbc48, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x729545ffbbc8, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893e9b in __pthread_cond_wait_common (abstime=0x729545ffbbc8, clockid=1, mutex=0x729534bc6cd0, cond=0x729545ffbc20) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x729545ffbc20, mutex=mutex@entry=0x729534bc6cd0, abstime=abstime@entry=0x729545ffbbc8) at ./nptl/pthread_cond_wait.c:652
#5  0x00007295514b3ad3 in mono_os_cond_timedwait (cond=cond@entry=0x729545ffbc20, mutex=mutex@entry=0x729534bc6cd0, timeout_ms=timeout_ms@entry=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#6  0x00007295514b81a9 in mono_coop_cond_timedwait (cond=0x729545ffbc20, mutex=0x729534bc6cd0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/../../mono/utils/mono-coop-mutex.h:103
#7  mono_lifo_semaphore_timed_wait (semaphore=0x729534bc6cd0, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:55
#8  0x000072955141146f in do_icall (sig=0x7295300fb750, op=MINT_ICALLSIG_PP_P, ret_sp=0x729523b6b270, sp=0x729523b6b280, ptr=0x72955154c830 <ves_icall_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2451
#9  0x000072955140f9fa in do_icall_wrapper (frame=0x729545ffda80, sig=0x7295300fb750, op=MINT_ICALLSIG_PP_P, ret_sp=0x729523b6b270, sp=0x729523b6b280, ptr=0x72955154c830 <ves_icall_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal>, save_last_error=0, gc_transitions=0x729545ffdc6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2534
#10 0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729545ffdd20, context=context@entry=0x72953800c4f0, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#11 0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729545ffddd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#12 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509bbc80, params=0x0, exc=0x0, error=0x729545ffddd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#13 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509bbc80, params=params@entry=0x0, error=error@entry=0x729545ffddd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#14 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#15 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#16 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#17 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7295461ff640 (LWP 10718) ".NET Long Runni"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295343faa50) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7295343faa50) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7295343faa50, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7295343faa00, cond=0x7295343faa28) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7295343faa28, mutex=0x7295343faa00) at ./nptl/pthread_cond_wait.c:627
#5  0x00007295511cb130 in SystemNative_LowLevelMonitor_Wait (monitor=0x7295343faa00) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:153
#6  0x0000729551411412 in do_icall (sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x729523cef500, sp=0x729523cef510, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2433
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x7295461fe8d0, sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x729523cef500, sp=0x729523cef510, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0, gc_transitions=0x7295461fec6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x7295461fed20, context=context@entry=0x72953429d330, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x7295461fedd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x729550ab43a0, params=0x0, exc=0x0, error=0x7295461fedd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x729550ab43a0, params=params@entry=0x0, error=error@entry=0x7295461fedd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#13 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#14 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7295443c4640 (LWP 10717) ".NET TP Worker"):
#0  __futex_abstimed_wait_common64 (private=885317600, cancel=true, abstime=0x7295443c28f8, op=137, expected=0, futex_word=0x72951c099f00) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=885317600, abstime=0x7295443c28f8, clockid=1010580540, expected=0, futex_word=0x72951c099f00) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x72951c099f00, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7295443c28f8, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893e9b in __pthread_cond_wait_common (abstime=0x7295443c28f8, clockid=1, mutex=0x72951c099eb0, cond=0x72951c099ed8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x72951c099ed8, mutex=0x72951c099eb0, abstime=0x7295443c28f8) at ./nptl/pthread_cond_wait.c:652
#5  0x00007295511cb24d in SystemNative_LowLevelMonitor_TimedWait (monitor=0x72951c099eb0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:193
#6  0x000072955141146f in do_icall (sig=0x7295349f1dd8, op=MINT_ICALLSIG_PP_P, ret_sp=0x7295440c3520, sp=0x7295440c3530, ptr=0x7295511cb170 <SystemNative_LowLevelMonitor_TimedWait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2451
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x7295443c38a0, sig=0x7295349f1dd8, op=MINT_ICALLSIG_PP_P, ret_sp=0x7295440c3520, sp=0x7295440c3530, ptr=0x7295511cb170 <SystemNative_LowLevelMonitor_TimedWait>, save_last_error=0, gc_transitions=0x7295443c3c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x7295443c3d20, context=context@entry=0x72951c002680, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x7295443c3dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509dfa50, params=0x0, exc=0x0, error=0x7295443c3dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509dfa50, params=params@entry=0x0, error=error@entry=0x7295443c3dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#13 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#14 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7295446d2640 (LWP 10716) ".NET Sockets"):
#0  0x0000729551925e2e in epoll_wait (epfd=9, events=0x72953004cd80, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007295511c5de9 in WaitForSocketEventsInner (port=9, buffer=0x72953004cd80, count=0x7295443d10f0) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3164
#2  0x00007295511c5d20 in SystemNative_WaitForSocketEvents (port=9, buffer=0x72953004cd80, count=0x7295443d10f0) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3490
#3  0x0000729551411447 in do_icall (sig=0x729530095918, op=MINT_ICALLSIG_PPP_P, ret_sp=0x7295443d11b0, sp=0x7295443d11c0, ptr=0x7295511c5cd0 <SystemNative_WaitForSocketEvents>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2463
#4  0x000072955140f9b6 in do_icall_wrapper (frame=0x7295446d1ab0, sig=0x729530095918, op=MINT_ICALLSIG_PPP_P, ret_sp=0x7295443d11b0, sp=0x7295443d11c0, ptr=0x7295511c5cd0 <SystemNative_WaitForSocketEvents>, save_last_error=0, gc_transitions=0x7295446d1c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#5  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x7295446d1d20, context=context@entry=0x729528002910, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#6  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x7295446d1dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#7  0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509dbc40, params=0x0, exc=0x0, error=0x7295446d1dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#8  mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509dbc40, params=params@entry=0x0, error=error@entry=0x7295446d1dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#9  0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#10 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#11 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x72954487b640 (LWP 10715) ".NET TP Gate"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295240090f0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7295240090f0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7295240090f0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7295240090a0, cond=0x7295240090c8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7295240090c8, mutex=0x7295240090a0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007295511cb130 in SystemNative_LowLevelMonitor_Wait (monitor=0x7295240090a0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:153
#6  0x0000729551411412 in do_icall (sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x72954473a460, sp=0x72954473a470, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2433
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x72954487a9f0, sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x72954473a460, sp=0x72954473a470, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0, gc_transitions=0x72954487ac6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x72954487ad20, context=context@entry=0x729524002680, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x72954487add8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509d7710, params=0x0, exc=0x0, error=0x72954487add8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509d7710, params=params@entry=0x0, error=error@entry=0x72954487add8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#13 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#14 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x729544ed9640 (LWP 10713) ".NET Timer"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x72952c0083b0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x72952c0083b0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x72952c0083b0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x72952c008360, cond=0x72952c008388) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x72952c008388, mutex=0x72952c008360) at ./nptl/pthread_cond_wait.c:627
#5  0x00007295511cb130 in SystemNative_LowLevelMonitor_Wait (monitor=0x72952c008360) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:153
#6  0x0000729551411412 in do_icall (sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x729544bd8430, sp=0x729544bd8440, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2433
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x729544ed89f0, sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x729544bd8430, sp=0x729544bd8440, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0, gc_transitions=0x729544ed8c6c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x729544ed8d20, context=context@entry=0x72952c002680, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x0, exc=<optimized out>, error=0x729544ed8dd8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x60517f02bfa0, obj=0x7295509c14a8, params=0x0, exc=0x0, error=0x729544ed8dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=0x60517f02bfa0, obj=obj@entry=0x7295509c14a8, params=params@entry=0x0, error=error@entry=0x729544ed8dd8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955154ca79 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1219
#13 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#14 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x72954c0dd640 (LWP 10707) ".NET SigHandler"):
#0  __GI___libc_read (nbytes=1, buf=0x72954c0dce47, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=7, buf=0x72954c0dce47, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007295511ca94d in SignalHandlerLoop (arg=0x60517ee3c2d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:331
#3  0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x72954e1fe640 (LWP 10706) "Finalizer"):
#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7295516030c0 <finalizer_sem>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x7295516030c0 <finalizer_sem>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7295516030c0 <finalizer_sem>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139
#3  0x000072955189cbdf in do_futex_wait (sem=sem@entry=0x7295516030c0 <finalizer_sem>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4  0x000072955189cc78 in __new_sem_wait_slow64 (sem=0x7295516030c0 <finalizer_sem>, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5  0x000072955189ccf1 in __new_sem_wait (sem=<optimized out>) at ./nptl/sem_wait.c:42
#6  0x0000729551574998 in mono_os_sem_wait (sem=<optimized out>, flags=<optimized out>) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/mono-os-semaphore.h:204
#7  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=<optimized out>) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/mono-coop-semaphore.h:41
#8  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:864
#9  0x000072955154c9ce in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1208
#10 start_wrapper (data=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1276
#11 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x72954e3ff640 (LWP 10705) ".NET EventPipe"):
#0  0x0000729551918bcf in __GI___poll (fds=fds@entry=0x729548002970, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000072955148a3cd in ipc_poll_fds (fds=0x729548002970, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:489
#2  ds_ipc_poll (poll_handles_data=0x729548002760, poll_handles_data_len=1, timeout_ms=timeout_ms@entry=4294967295, callback=callback@entry=0x729551490740 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1124
#3  0x000072955148a009 in ds_ipc_stream_factory_get_next_available_stream (callback=callback@entry=0x729551490740 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:393
#4  0x000072955148dd27 in server_loop_tick (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:118
#5  server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:185
#6  0x0000729551490721 in ep_rt_thread_mono_start_func (data=0x60517e172070) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:942
#7  0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7295507ff640 (LWP 10704) "SGen worker"):
#0  0x00007295518ea42f in __GI___wait4 (pid=pid@entry=11138, stat_loc=stat_loc@entry=0x7295507fd280, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1  0x00007295518ea3ab in __GI___waitpid (pid=pid@entry=11138, stat_loc=stat_loc@entry=0x7295507fd280, options=options@entry=0) at ./posix/waitpid.c:38
#2  0x00007295513fd290 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:877
#3  mono_dump_native_crash_info (signal=<optimized out>, mctx=mctx@entry=0x7295507fddf0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:899
#4  0x00007295513a2d6e in mono_handle_native_crash (signal=0x729551254207 "SIGSEGV", mctx=0x7295507fddf0, info=0x7295507fe0b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2999
#5  0x0000729551307286 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7295507fe0b0, context=0x7295507fdf80, debug_fault_addr=0x8) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3842
#6  <signal handler called>
#7  sgen_safe_object_is_small (obj=0x729523f13a60, type=<optimized out>) at /__w/1/s/src/mono/mono/sgen/../../mono/sgen/sgen-gc.h:828
#8  major_scan_object_concurrent_with_evacuation (full_object=0x7295457daf70, desc=<optimized out>, queue=queue@entry=0x729552024010) at /__w/1/s/src/mono/mono/sgen/sgen-scan-object.h:66
#9  0x00007295515c628a in drain_gray_stack_concurrent_with_evacuation (queue=0x729552024010) at /__w/1/s/src/mono/mono/sgen/sgen-marksweep-drain-gray-stack.h:347
#10 drain_gray_stack_concurrent (queue=0x729552024010) at /__w/1/s/src/mono/mono/sgen/sgen-marksweep.c:1310
#11 0x00007295515e61f8 in marker_idle_func (data_untyped=0x729552024008) at /__w/1/s/src/mono/mono/sgen/sgen-workers.c:363
#12 0x00007295515e51ac in thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:219
#13 0x0000729551894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x0000729551926850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x729551f08740 (LWP 10703) "dotnet"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x60517f24f984) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x60517f24f984) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x60517f24f984, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x0000729551893a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x60517f24f930, cond=0x60517f24f958) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x60517f24f958, mutex=0x60517f24f930) at ./nptl/pthread_cond_wait.c:627
#5  0x00007295511cb130 in SystemNative_LowLevelMonitor_Wait (monitor=0x60517f24f930) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:153
#6  0x0000729551411412 in do_icall (sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x7295516ff620, sp=0x7295516ff630, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2433
#7  0x000072955140f9b6 in do_icall_wrapper (frame=0x7ffdd5fbfd30, sig=0x729534013898, op=MINT_ICALLSIG_P_V, ret_sp=0x7295516ff620, sp=0x7295516ff630, ptr=0x7295511cb0e0 <SystemNative_LowLevelMonitor_Wait>, save_last_error=0, gc_transitions=0x7ffdd5fc003c) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2530
#8  0x0000729551400e68 in mono_interp_exec_method (frame=<optimized out>, frame@entry=0x7ffdd5fc00f0, context=context@entry=0x60517e177c70, clause_args=<optimized out>, clause_args@entry=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:4396
#9  0x00007295513fe7e6 in interp_runtime_invoke (method=<optimized out>, obj=<optimized out>, params=0x7ffdd5fc0190, exc=<optimized out>, error=0x7ffdd5fc01d8) at /__w/1/s/src/mono/mono/mini/interp/interp.c:2271
#10 0x0000729551534927 in do_runtime_invoke (method=0x60517e2573c8, obj=0x0, params=0x7ffdd5fc0190, exc=0x0, error=0x7ffdd5fc01d8) at /__w/1/s/src/mono/mono/metadata/object.c:2576
#11 mono_runtime_invoke_checked (method=method@entry=0x60517e2573c8, obj=obj@entry=0x0, params=params@entry=0x7ffdd5fc0190, error=error@entry=0x7ffdd5fc01d8) at /__w/1/s/src/mono/mono/metadata/object.c:2792
#12 0x000072955153b6e8 in do_exec_main_checked (method=0x60517e2573c8, args=<optimized out>, error=0x7ffdd5fc01d8) at /__w/1/s/src/mono/mono/metadata/object.c:4706
#13 mono_runtime_exec_main_checked (method=0x60517e2573c8, args=<optimized out>, error=0x7ffdd5fc01d8) at /__w/1/s/src/mono/mono/metadata/object.c:4812
#14 0x000072955136190f in mono_jit_exec_internal (assembly=<optimized out>, domain=<optimized out>, argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:1366
#15 mono_jit_exec (domain=<optimized out>, assembly=<optimized out>, argc=12, argv=0x60517e0cc080) at /__w/1/s/src/mono/mono/mini/driver.c:1311
#16 0x000072955136417d in mono_main (argc=14, argv=0x60517e0cc070) at /__w/1/s/src/mono/mono/mini/driver.c:2587
#17 0x00007295513da477 in monovm_execute_assembly (argc=<optimized out>, argv=0x60517e0c16c0, managedAssemblyPath=0x60517e0cf8f0 "/datadisks/disk1/work/9D82087C/w/AC520976/e/xunit.console.dll", exitCode=0x7ffdd5fc054c) at /__w/1/s/src/mono/mono/mini/monovm.c:313
#18 0x0000729551ba7ea1 in coreclr_t::execute_assembly (this=0x60517e0c8920, argc=11, argv=0x60517e0c16c0, managed_assembly_path=0x60517e0cf8f0 "/datadisks/disk1/work/9D82087C/w/AC520976/e/xunit.console.dll", exit_code=0x7ffdd5fc054c) at /__w/1/s/src/native/corehost/hostpolicy/coreclr.cpp:108
#19 0x0000729551bd38a9 in run_app_for_context (context=..., argc=11, argv=0x60517e0b88d0) at /__w/1/s/src/native/corehost/hostpolicy/hostpolicy.cpp:250
#20 0x0000729551bd3b65 in run_app (argc=11, argv=0x60517e0b88d0) at /__w/1/s/src/native/corehost/hostpolicy/hostpolicy.cpp:285
#21 0x0000729551bd49a4 in corehost_main (argc=13, argv=0x60517e0b88c0) at /__w/1/s/src/native/corehost/hostpolicy/hostpolicy.cpp:426
#22 0x0000729551ecd547 in ?? () from /datadisks/disk1/work/9D82087C/p/host/fxr/10.0.0/libhostfxr.so
#23 0x0000729551ec9748 in ?? () from /datadisks/disk1/work/9D82087C/p/host/fxr/10.0.0/libhostfxr.so
#24 0x0000729551ec60ed in ?? () from /datadisks/disk1/work/9D82087C/p/host/fxr/10.0.0/libhostfxr.so
#25 0x0000729551ec51e6 in ?? () from /datadisks/disk1/work/9D82087C/p/host/fxr/10.0.0/libhostfxr.so
#26 0x0000729551ebaa49 in hostfxr_main_startupinfo () from /datadisks/disk1/work/9D82087C/p/host/fxr/10.0.0/libhostfxr.so
#27 0x000060514604fef4 in ?? ()
#28 0x00006051460503c4 in ?? ()
#29 0x0000729551829d90 in __libc_start_call_main (main=main@entry=0x6051460502f0, argc=argc@entry=18, argv=argv@entry=0x7ffdd5fc1108) at ../sysdeps/nptl/libc_start_call_main.h:58
#30 0x0000729551829e40 in __libc_start_main_impl (main=0x6051460502f0, argc=18, argv=0x7ffdd5fc1108, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdd5fc10f8) at ../csu/libc-start.c:392
#31 0x000060514604ef0a in ?? ()
[Inferior 1 (process 10703) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7295515c294a):0x7295515c293a  08 00 74 70 4c 21 e0 48 39 c8 74 68 4d 8b 7d 00  ..tpL!.H9.thM.}.
0x7295515c294a  41 f6 47 08 04 74 34 41 f6 c7 01 0f 94 c0 4c 89  A.G..t4A......L.
0x7295515c295a  ff 48 83 e7 f8 0f 94 c1 08 c1 75 07 48 8b 3f 48  .H........u.H.?H
0x7295515c296a  83 e7 f8 4c 89 ee e8 4b 08 01 00 48 83 c0 07 48  ...L...K...H...H
./RunTests.sh: line 176: 10703 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Tests.deps.json xunit.console.dll System.Security.Cryptography.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/9D82087C/w/AC520976/e
----- end Fri Jul 25 06:56:47 PM UTC 2025 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

Where nothing looks like either OpenSSL or the crypto shim.

I haven't tried slapping SkipOnMono on the tests that would exercise this code. But I have tried over the last couple of weeks a few "time has passed, maybe this was a bug already fixed in main" merges from main, always yielding the same mysterious mono-only failures.

@bartonjs bartonjs mentioned this pull request Jul 31, 2025
@bartonjs
Copy link
Member Author

bartonjs commented Aug 5, 2025

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vcsjones
Copy link
Member

vcsjones commented Aug 5, 2025

@bartonjs I will try again to figure this one out now that I have some vacation rest in me.

@vcsjones
Copy link
Member

vcsjones commented Aug 7, 2025

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vcsjones
Copy link
Member

vcsjones commented Aug 7, 2025

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vcsjones vcsjones added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 7, 2025
@vcsjones
Copy link
Member

vcsjones commented Aug 7, 2025

Symbolicated stack:

 thread #2, name = 'SGen worker', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x8)
  * frame #0: 0x0000fffff78222d8 libcoreclr.so`sgen_vtable_get_descriptor(vtable=0x0000000000000000) at sgen-client-mono.h:29:33
    frame #1: 0x0000fffff782d148 libcoreclr.so`major_scan_object_concurrent_no_evacuation [inlined] major_copy_or_mark_object_concurrent_no_evacuation(ptr=0x0000ffffe65269e0, obj=0x0000ffffe65fab30, queue=0x0000fffff7ff5010) at sgen-marksweep-drain-gray-stack.h:158:10
    frame #2: 0x0000fffff782d0e0 libcoreclr.so`major_scan_object_concurrent_no_evacuation(full_object=0x0000ffffe65269d0, desc=250, queue=0x0000fffff7ff5010) at sgen-scan-object.h:66:3
    frame #3: 0x0000fffff782b764 libcoreclr.so`drain_gray_stack_concurrent_no_evacuation(queue=0x0000fffff7ff5010) at sgen-marksweep-drain-gray-stack.h:347:3
    frame #4: 0x0000fffff780b95c libcoreclr.so`drain_gray_stack_concurrent(queue=0x0000fffff7ff5010) at sgen-marksweep.c:1312:10
    frame #5: 0x0000fffff77dd35c libcoreclr.so`sgen_drain_gray_stack(ctx=ScanCopyContext @ 0x0000fffff6bfe840) at sgen-gc.c:578:9
    frame #6: 0x0000fffff785a6ec libcoreclr.so`marker_idle_func(data_untyped=0x0000fffff7ff5008) at sgen-workers.c:363:3
    frame #7: 0x0000fffff78590a4 libcoreclr.so`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:219:5
    frame #8: 0x0000fffff7ab0eb4 libc.so.6`start_thread + 676
    frame #9: 0x0000fffff7b171cc libc.so.6`thread_start + 12

@vcsjones
Copy link
Member

vcsjones commented Aug 7, 2025

While I can solidly reproduce this issue now, I don't really know how to proceed with it. It seems very much likely there is something mono specific going on here.

Some notes.

This does not reproduce with dotnet that gets installed in the .dotnet directory. You need to use the testhost that gets built.

Steps to reproduce:

  1. Jump in a container

    container run -it mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-arm64v8

    If you are on AMD, use the AMD one. This example is also using Apple Container. Swap for docker if you need to.

  2. Follow my "Prerequisites" section for Azure Linux. https://gist.github.com/vcsjones/6c0bddcc6bbe721a5f43673359552f67#prerequisites

    Note that the container images uses tdnf, not dnf. So do all your stuff with tdnf.

  3. Build with ./build.sh -ci -s mono+libs+libs.tests -rc Checked -c Debug /p:BundleXunitRunner=true

  4. Change working directory to artifacts/bin/System.Security.Cryptography.Tests/Debug/net10.0-unix

  5. Run something like /source/runtime/artifacts/bin/testhost/net10.0-linux-Debug-arm64/dotnet exec --runtimeconfig System.Security.Cryptography.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Tests.deps.json xunit.console.dll System.Security.Cryptography.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing

    Change the initial path for the test host if you checked it out somewhere other than /source/runtime.

@bartonjs bartonjs modified the milestones: 10.0.0, 11.0.0 Aug 25, 2025
@bartonjs
Copy link
Member Author

/azp run runtime-libraries-coreclr outerloop

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-System.Security NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add RFC5649 KeyWrap
2 participants