net7.0 app crashes in ProfilerAddNewRegion after profiler attach

[k15tfu]

Hi!

I faced with the following crash in .NET 7 app on Windows x64 after attaching the profiler:

Exception thrown at 0x00007FFD6979E634 (coreclr.dll) in ....exe: 0xC0000005: Access violation reading location 0x0000000000000028.

 	coreclr.dll!CrstBase::Enter() Line 292	C++
>	[Inline Frame] coreclr.dll!CrstBase::AcquireLock(CrstBase *) Line 187	C++
 	[Inline Frame] coreclr.dll!CrstBase::CrstHolder::{ctor}(CrstBase *) Line 378	C++
 	coreclr.dll!GenerationTable::AddRecord(int generation=0x00000000, unsigned char * rangeStart=0x0000022fb8000020, unsigned char * rangeEnd=0x0000022fb8000020, unsigned char * rangeEndReserved=0x0000022fb8400000) Line 766	C++
 	[Inline Frame] coreclr.dll!ProfilerAddNewRegion(int) Line 968	C++
 	[Inline Frame] coreclr.dll!GCToEEInterface::DiagAddNewRegion(int generation, unsigned char * rangeStart, unsigned char * rangeEnd, unsigned char * rangeEndReserved) Line 1786	C++
 	coreclr.dll!SVR::gc_heap::soh_try_fit(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007, int * commit_failed_p=0x000000cc4a97bec4, int * short_seg_end_p=0x0000000000000000) Line 17015	C++
 	coreclr.dll!SVR::gc_heap::allocate_soh(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007) Line 17100	C++
 	coreclr.dll!SVR::gc_heap::try_allocate_more_space(alloc_context * acontext=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000, int gen_number=0x00000000) Line 18052	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate_more_space(alloc_context *) Line 18499	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate(unsigned __int64) Line 18555	C++
 	coreclr.dll!SVR::GCHeap::Alloc(gc_alloc_context * context=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000) Line 46244	C++
 	coreclr.dll!Alloc(unsigned __int64 size=0x0000000000000128, GC_ALLOC_FLAGS flags) Line 227	C++
 	coreclr.dll!AllocateString(unsigned long cchStringLength=0x00000089) Line 858	C++
 	coreclr.dll!FramedAllocateString(unsigned long stringLength=0x00000089) Line 2415	C++
 	00007ffd0fbef940()	Unknown

Here is disasm of the 1st frame:

   00007FFD6979E620  mov         qword ptr [rsp+8],rbx  
   00007FFD6979E625  mov         qword ptr [rsp+10h],rbp  
   00007FFD6979E62A  mov         qword ptr [rsp+18h],rsi  
   00007FFD6979E62F  push        rdi  
   00007FFD6979E630  sub         rsp,20h  
-> 00007FFD6979E634  test        dword ptr [rcx+28h],10Ch  -- rcx = 0x0000000000000000 
   00007FFD6979E63B  mov         rsi,rcx  
   00007FFD6979E63E  mov         edx,dword ptr [_tls_index (07FFD69C4F748h)]  
   00007FFD6979E644  mov         rax,qword ptr gs:[58h]  

.NET 7.0.0, or latest public .NET SDK 7.0.200-preview.22571.16 (https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200-preview.22571.16/dotnet-sdk-7.0.200-win-x64.zip)
Windows 10 21H2

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hi!

I faced with the following crash in .NET 7 app on Windows x64 after attaching the profiler:

Exception thrown at 0x00007FFD6979E634 (coreclr.dll) in ....exe: 0xC0000005: Access violation reading location 0x0000000000000028.

 	coreclr.dll!CrstBase::Enter() Line 292	C++
>	[Inline Frame] coreclr.dll!CrstBase::AcquireLock(CrstBase *) Line 187	C++
 	[Inline Frame] coreclr.dll!CrstBase::CrstHolder::{ctor}(CrstBase *) Line 378	C++
 	coreclr.dll!GenerationTable::AddRecord(int generation=0x00000000, unsigned char * rangeStart=0x0000022fb8000020, unsigned char * rangeEnd=0x0000022fb8000020, unsigned char * rangeEndReserved=0x0000022fb8400000) Line 766	C++
 	[Inline Frame] coreclr.dll!ProfilerAddNewRegion(int) Line 968	C++
 	[Inline Frame] coreclr.dll!GCToEEInterface::DiagAddNewRegion(int generation, unsigned char * rangeStart, unsigned char * rangeEnd, unsigned char * rangeEndReserved) Line 1786	C++
 	coreclr.dll!SVR::gc_heap::soh_try_fit(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007, int * commit_failed_p=0x000000cc4a97bec4, int * short_seg_end_p=0x0000000000000000) Line 17015	C++
 	coreclr.dll!SVR::gc_heap::allocate_soh(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007) Line 17100	C++
 	coreclr.dll!SVR::gc_heap::try_allocate_more_space(alloc_context * acontext=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000, int gen_number=0x00000000) Line 18052	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate_more_space(alloc_context *) Line 18499	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate(unsigned __int64) Line 18555	C++
 	coreclr.dll!SVR::GCHeap::Alloc(gc_alloc_context * context=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000) Line 46244	C++
 	coreclr.dll!Alloc(unsigned __int64 size=0x0000000000000128, GC_ALLOC_FLAGS flags) Line 227	C++
 	coreclr.dll!AllocateString(unsigned long cchStringLength=0x00000089) Line 858	C++
 	coreclr.dll!FramedAllocateString(unsigned long stringLength=0x00000089) Line 2415	C++
 	00007ffd0fbef940()	Unknown

Here is disasm of the 1st frame:

   00007FFD6979E620  mov         qword ptr [rsp+8],rbx  
   00007FFD6979E625  mov         qword ptr [rsp+10h],rbp  
   00007FFD6979E62A  mov         qword ptr [rsp+18h],rsi  
   00007FFD6979E62F  push        rdi  
   00007FFD6979E630  sub         rsp,20h  
-> 00007FFD6979E634  test        dword ptr [rcx+28h],10Ch  -- rcx = 0x0000000000000000 
   00007FFD6979E63B  mov         rsi,rcx  
   00007FFD6979E63E  mov         edx,dword ptr [_tls_index (07FFD69C4F748h)]  
   00007FFD6979E644  mov         rax,qword ptr gs:[58h]  

.NET 7.0.0
Windows 10 21H2

Author:	k15tfu
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hi!

I faced with the following crash in .NET 7 app on Windows x64 after attaching the profiler:

Exception thrown at 0x00007FFD6979E634 (coreclr.dll) in ....exe: 0xC0000005: Access violation reading location 0x0000000000000028.

 	coreclr.dll!CrstBase::Enter() Line 292	C++
>	[Inline Frame] coreclr.dll!CrstBase::AcquireLock(CrstBase *) Line 187	C++
 	[Inline Frame] coreclr.dll!CrstBase::CrstHolder::{ctor}(CrstBase *) Line 378	C++
 	coreclr.dll!GenerationTable::AddRecord(int generation=0x00000000, unsigned char * rangeStart=0x0000022fb8000020, unsigned char * rangeEnd=0x0000022fb8000020, unsigned char * rangeEndReserved=0x0000022fb8400000) Line 766	C++
 	[Inline Frame] coreclr.dll!ProfilerAddNewRegion(int) Line 968	C++
 	[Inline Frame] coreclr.dll!GCToEEInterface::DiagAddNewRegion(int generation, unsigned char * rangeStart, unsigned char * rangeEnd, unsigned char * rangeEndReserved) Line 1786	C++
 	coreclr.dll!SVR::gc_heap::soh_try_fit(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007, int * commit_failed_p=0x000000cc4a97bec4, int * short_seg_end_p=0x0000000000000000) Line 17015	C++
 	coreclr.dll!SVR::gc_heap::allocate_soh(int gen_number=0x00000000, unsigned __int64 size=0x0000000000000128, alloc_context * acontext=0x0000027024fb1418, unsigned int flags=0x00000000, int align_const=0x00000007) Line 17100	C++
 	coreclr.dll!SVR::gc_heap::try_allocate_more_space(alloc_context * acontext=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000, int gen_number=0x00000000) Line 18052	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate_more_space(alloc_context *) Line 18499	C++
 	[Inline Frame] coreclr.dll!SVR::gc_heap::allocate(unsigned __int64) Line 18555	C++
 	coreclr.dll!SVR::GCHeap::Alloc(gc_alloc_context * context=0x0000027024fb1418, unsigned __int64 size=0x0000000000000128, unsigned int flags=0x00000000) Line 46244	C++
 	coreclr.dll!Alloc(unsigned __int64 size=0x0000000000000128, GC_ALLOC_FLAGS flags) Line 227	C++
 	coreclr.dll!AllocateString(unsigned long cchStringLength=0x00000089) Line 858	C++
 	coreclr.dll!FramedAllocateString(unsigned long stringLength=0x00000089) Line 2415	C++
 	00007ffd0fbef940()	Unknown

Here is disasm of the 1st frame:

   00007FFD6979E620  mov         qword ptr [rsp+8],rbx  
   00007FFD6979E625  mov         qword ptr [rsp+10h],rbp  
   00007FFD6979E62A  mov         qword ptr [rsp+18h],rsi  
   00007FFD6979E62F  push        rdi  
   00007FFD6979E630  sub         rsp,20h  
-> 00007FFD6979E634  test        dword ptr [rcx+28h],10Ch  -- rcx = 0x0000000000000000 
   00007FFD6979E63B  mov         rsi,rcx  
   00007FFD6979E63E  mov         edx,dword ptr [_tls_index (07FFD69C4F748h)]  
   00007FFD6979E644  mov         rax,qword ptr gs:[58h]  

.NET 7.0.0, or latest public .NET SDK 7.0.200-preview.22571.16 (https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200-preview.22571.16/dotnet-sdk-7.0.200-win-x64.zip)
Windows 10 21H2

Author:	k15tfu
Assignees:	-
Labels:	area-CodeGen-coreclr, area-GC-coreclr, untriaged
Milestone:	-

[jkotas]

Related change: #57101

[mangod9]

Thanks for reporting this @k15tfu -- does it fail consistently for your app when attaching a profiler?

@cshung could you please take a look?

[ww898]

I recently had the same issue with .NET 7.0:

0:041> k
 # Child-SP          RetAddr               Call Site
00 000000e9`dc17eca0 00007fff`1629302b     coreclr!CrstBase::Enter+0x2b [D:\a\_work\1\s\src\coreclr\vm\crst.cpp @ 292] 
01 (Inline Function) --------`--------     coreclr!CrstBase::AcquireLock+0x8 [D:\a\_work\1\s\src\coreclr\vm\crst.h @ 187] 
02 (Inline Function) --------`--------     coreclr!CrstBase::CrstHolder::{ctor}+0xc [D:\a\_work\1\s\src\coreclr\vm\crst.h @ 378] 
03 000000e9`dc17ecd0 00007fff`16135b7e     coreclr!GenerationTable::AddRecord+0x2f [D:\a\_work\1\s\src\coreclr\vm\proftoeeinterfaceimpl.cpp @ 766] 
04 000000e9`dc17ed20 00007fff`160ec898     coreclr!ProfilerAddNewRegion+0x56 [D:\a\_work\1\s\src\coreclr\vm\proftoeeinterfaceimpl.cpp @ 968] 
05 (Inline Function) --------`--------     coreclr!GCToEEInterface::DiagAddNewRegion+0x10 [D:\a\_work\1\s\src\coreclr\vm\gcenv.ee.cpp @ 1786] 
06 (Inline Function) --------`--------     coreclr!SVR::gc_heap::soh_try_fit+0x228 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 16811] 
07 000000e9`dc17ed60 00007fff`1617f5c8     coreclr!SVR::gc_heap::allocate_soh+0x2b8 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 16893] 
08 000000e9`dc17ee50 00007fff`160e9b64     coreclr!SVR::gc_heap::try_allocate_more_space+0x95928 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 17848] 
09 000000e9`dc17eeb0 00007fff`160ec097     coreclr!SVR::gc_heap::allocate_more_space+0x5c [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 18323] 
0a (Inline Function) --------`--------     coreclr!SVR::gc_heap::allocate+0x7c [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 18351] 
0b 000000e9`dc17ef00 00007fff`160099ae     coreclr!SVR::GCHeap::Alloc+0xb7 [D:\a\_work\1\s\src\coreclr\gc\gc.cpp @ 45892] 
0c (Inline Function) --------`--------     coreclr!Alloc+0xb7 [D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp @ 226] 
0d (Inline Function) --------`--------     coreclr!AllocateObject+0x11e [D:\a\_work\1\s\src\coreclr\vm\gchelpers.cpp @ 973] 
0e 000000e9`dc17ef50 00007ffe`bc6ba67a     coreclr!JIT_New+0x1fe [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 2310] 
0f 000000e9`dc17f1d0 00007ffe`bc6b45ec     0x00007ffe`bc6ba67a
10 000000e9`dc17f220 00007ffe`bc6a7ed8     0x00007ffe`bc6b45ec
11 000000e9`dc17f270 00007ffe`bc6ab268     0x00007ffe`bc6a7ed8
12 000000e9`dc17f2f0 00007ffe`bc6b4419     0x00007ffe`bc6ab268
13 000000e9`dc17f3b0 00007ffe`bc6b4303     0x00007ffe`bc6b4419
14 000000e9`dc17f3f0 00007ffe`bc6b41b9     0x00007ffe`bc6b4303
15 000000e9`dc17f430 00007ffe`bc6b3ffe     0x00007ffe`bc6b41b9
16 000000e9`dc17f480 00007ffe`bc6b39a9     0x00007ffe`bc6b3ffe
17 000000e9`dc17f500 00007ffe`bc6b3810     0x00007ffe`bc6b39a9
18 000000e9`dc17f5e0 00007ffe`bc6b36b2     0x00007ffe`bc6b3810
19 000000e9`dc17f620 00007ffe`bc6b296f     0x00007ffe`bc6b36b2
1a 000000e9`dc17f670 00007ffe`bc6af91a     0x00007ffe`bc6b296f
1b 000000e9`dc17f6e0 00007ffe`bc6a9a1e     0x00007ffe`bc6af91a
1c 000000e9`dc17f770 00007ffe`b7af83ab     0x00007ffe`bc6a9a1e
1d 000000e9`dc17f7f0 00007ffe`b66ae8ca     0x00007ffe`b7af83ab
1e 000000e9`dc17f940 00007ffe`b66ae80b     0x00007ffe`b66ae8ca
1f 000000e9`dc17f990 00007ffe`b66ae756     0x00007ffe`b66ae80b
20 000000e9`dc17f9d0 00007fff`1614dbb3     0x00007ffe`b66ae756
21 000000e9`dc17fa10 00007fff`1601872c     coreclr!CallDescrWorkerInternal+0x83
22 000000e9`dc17fa50 00007fff`1613dac3     coreclr!DispatchCallSimple+0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 221] 
23 000000e9`dc17fae0 00007fff`160de1a1     coreclr!ThreadNative::KickOffThread_Worker+0x63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158] 
24 (Inline Function) --------`--------     coreclr!ManagedThreadBase_DispatchInner+0xd [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7298] 
25 000000e9`dc17fb40 00007fff`160de0b7     coreclr!ManagedThreadBase_DispatchMiddle+0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7342] 
26 000000e9`dc17fc20 00007fff`160ddfa9     coreclr!ManagedThreadBase_DispatchOuter+0xab [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7501] 
27 (Inline Function) --------`--------     coreclr!ManagedThreadBase_FullTransition+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7546] 
28 (Inline Function) --------`--------     coreclr!ManagedThreadBase::KickOff+0x2d [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7581] 
29 000000e9`dc17fcc0 00007fff`685074b4     coreclr!ThreadNative::KickOffThread+0x79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230] 
2a 000000e9`dc17fd20 00007fff`69e426a1     KERNEL32!BaseThreadInitThunk+0x14
2b 000000e9`dc17fd50 00000000`00000000     ntdll!RtlUserThreadStart+0x21

[cshung]

How exactly are we attaching the profiler? I am unable to reproduce it by the following procedure:

set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER={55b9554d-6115-45a2-be1e-c80f7fa35369}
set CORECLR_PROFILER_PATH=C:\dev\runtime\src\tests\profiler\native\build\Debug\Profiler.dll
... launch the process ...

With a properly constructed GenerationTable, the critical section should be initialized as part of its only constructor.

runtime/src/coreclr/vm/proftoeeinterfaceimpl.cpp

Line 742 in a230293

GenerationTable::GenerationTable() : mutex(CrstLeafLock, CRST_UNSAFE_ANYMODE)

I am suspecting we might be skipping the constructor because of this line.

runtime/src/coreclr/vm/proftoeeinterfaceimpl.cpp

Line 904 in a230293

static GenerationTable *s_currentGenerationTable;

Without an explicit initialization of this to nullptr, we might be failing to hit the initialization branch and thus skipped the initialization.

It is trivial to add the explicit initialization there, but it would be really nice if I can repro the bug and validate that does fix the issue.

[k15tfu]

@mangod9 I can reproduce this 10 out of 10 times.

[ww898]

Can I expect that the fix will backported to .NET 7.0.1 ? Attach crashes are very painful for our customers.