[libraries-pgo] SIGSEGV in libclrjit.so!Compiler::optMarkLoopRemoved(unsigned int)

[rzikm]

We have a few crashes on the runtime-coreclr libraries-pgo leg on Ubuntu 18.04, both ARM and x64.

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9c87c5c4deb0499081/System.Net.Http.Unit.Tests/1/console.5bbac1e3.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d9d770df61c8470993/System.Net.Http.Unit.Tests/1/console.a9395c58.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-62ca6e33b69446b7b9/System.Net.Http.Unit.Tests/1/console.78831c68.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-07b013262b854ee2b3/System.Net.Http.Unit.Tests/1/console.c435ec1c.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9d085c25e76846d882/System.Net.Http.Unit.Tests/1/console.ceac850a.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d09ca73c69e646cfbe/System.Net.Http.Unit.Tests/1/console.c9863e43.log?helixlogtype=result

I checked the crash dump for the first one:

lldb dumpstack

(lldb) dumpstack
OS Thread Id: 0x14d7 (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libclrjit.so!Compiler::optMarkLoopRemoved(unsigned int) + 0xdd [/__w/1/s/src/coreclr/jit/optimizer.cpp:10093]
Child-SP         RetAddr          Caller, Callee
00007F89A080AC30 00007f89a0b51ea6 libclrjit.so!Compiler::optUpdateLoopsBeforeRemoveBlock(BasicBlock*, bool) + 0x206 [/__w/1/s/src/coreclr/jit/optimizer.cpp:437], calling libclrjit.so!Compiler::optMarkLoopRemoved(unsigned int) [/__w/1/s/src/coreclr/jit/optimizer.cpp:10085]
00007F89A080ACB0 00007f89a09e155e libclrjit.so!Compiler::fgUnreachableBlock(BasicBlock*) + 0x1ae [/__w/1/s/src/coreclr/jit/block.h:605], calling libclrjit.so!Compiler::optUpdateLoopsBeforeRemoveBlock(BasicBlock*, bool) [/__w/1/s/src/coreclr/jit/optimizer.cpp:394]
00007F89A080ACE0 00007f89a09c1f21 libclrjit.so!Compiler::fgRemoveBlock(BasicBlock*, bool) + 0x291 [/__w/1/s/src/coreclr/jit/fgbasic.cpp:4573], calling libclrjit.so!Compiler::fgUnreachableBlock(BasicBlock*) [/__w/1/s/src/coreclr/jit/fgopt.cpp:2423]
00007F89A080AD80 00007f89a09e6ab7 libclrjit.so!Compiler::fgUpdateFlowGraph(bool) + 0x9a7 [/__w/1/s/src/coreclr/jit/fgopt.cpp:0], calling libclrjit.so!Compiler::fgRemoveBlock(BasicBlock*, bool) [/__w/1/s/src/coreclr/jit/fgbasic.cpp:4540]
00007F89A080ADF0 00007f89a0b5acd4 libclrjit.so!Compiler::optOptimizeLayout() + 0x94 [/__w/1/s/src/coreclr/jit/optimizer.cpp:5164], calling libclrjit.so!Compiler::fgUpdateFlowGraph(bool) [/__w/1/s/src/coreclr/jit/fgopt.cpp:5876]
00007F89A080AE10 00007f89a0b6ada6 libclrjit.so!Phase::Run() + 0x76 [/__w/1/s/src/coreclr/jit/phase.cpp:61]
00007F89A080AE60 00007f89a0994edd libclrjit.so!Compiler::compCompile(void**, unsigned int*, JitFlags*) + 0x139d [/__w/1/s/src/coreclr/jit/phase.h:37], calling libclrjit.so!Phase::Run() [/__w/1/s/src/coreclr/jit/phase.cpp:58]
00007F89A080AF60 00007f89a0a9eadf libclrjit.so!Compiler::fgVerifyHandlerTab() + 0xf0f [/__w/1/s/src/coreclr/jit/jiteh.cpp:0], calling libclrjit.so!___lldb_unnamed_symbol8237
00007F89A080AFA0 00007f8a1ac7822c libcoreclr.so!CEEInfo::logMsg(unsigned int, char const*, __va_list_tag*) + 0x1c [/__w/1/s/src/coreclr/vm/jitinterface.cpp:10703], calling libcoreclr.so!LoggingOn(unsigned int, unsigned int) [/__w/1/s/src/coreclr/utilcode/log.cpp:238]
00007F89A080B120 00007f89a0998f01 libclrjit.so!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) + 0xdb1 [/__w/1/s/src/coreclr/jit/compiler.hpp:3877], calling libclrjit.so!Compiler::compCompile(void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:4325]
00007F89A080B1D0 00007f89a0997277 libclrjit.so!Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*) + 0x8a7 [/__w/1/s/src/coreclr/jit/compiler.cpp:0], calling libclrjit.so!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:6292]
00007F89A080B200 00007f89a0971bd3 libclrjit.so!CodeGen::CodeGen(Compiler*) + 0x193 [/__w/1/s/src/coreclr/jit/jit.h:782], calling libclrjit.so!Compiler::compGetJitDefaultFill(Compiler*) [/__w/1/s/src/coreclr/jit/compiler.cpp:2054]
00007F89A080B240 00007f89a09719c3 libclrjit.so!getCodeGenerator(Compiler*) + 0x53 [/__w/1/s/src/coreclr/jit/codegencommon.cpp:62], calling libclrjit.so!CodeGen::CodeGen(Compiler*) [/__w/1/s/src/coreclr/jit/codegencommon.cpp:74]
00007F89A080B260 00007f89a098fafd libclrjit.so!Compiler::compInit(ArenaAllocator*, CORINFO_METHOD_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, InlineInfo*) + 0x4ed [/__w/1/s/src/coreclr/jit/jitstd/list.h:301], calling libclrjit.so!ArenaAllocator::getMemStatsAllocator(CompMemKind) [/__w/1/s/src/coreclr/jit/alloc.cpp:283]
00007F89A080B2B0 00007f89a0999e61 libclrjit.so!jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) + 0x341 [/__w/1/s/src/coreclr/jit/compiler.cpp:7375], calling libclrjit.so!Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:5475]
00007F89A080B3B0 00007f8a1ad28a67 libcoreclr.so!EETypeHashTable::CompareInstantiatedType(TypeHandle, Module*, unsigned int, Instantiation) + 0xd7 [/__w/1/s/src/coreclr/vm/typehandle.h:698], calling libcoreclr.so!TypeHandle::GetInstantiation() const [/__w/1/s/src/coreclr/vm/typehandle.cpp:361]
00007F89A080B8D0 00007f89a09a65c6 libclrjit.so!CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) + 0x116 [/__w/1/s/src/coreclr/jit/ee_il_dll.cpp:276], calling libclrjit.so!jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) [/__w/1/s/src/coreclr/jit/compiler.cpp:7261]
00007F89A080B970 00007f8a1ac7cede libcoreclr.so!invokeCompileMethodHelper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) + 0x1ae [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12379]
00007F89A080B980 00007f8a1aba656a libcoreclr.so!Thread::EnablePreemptiveGC() + 0x1a [/__w/1/s/src/coreclr/vm/threads.inl:42], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080B9C0 00007f8a1ac7cf7a libcoreclr.so!invokeCompileMethod(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) + 0x5a [/__w/1/s/src/coreclr/vm/jitinterface.cpp:0], calling libcoreclr.so!invokeCompileMethodHelper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12284]
00007F89A080BA30 00007f8a1ac7e02e libcoreclr.so!UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, CORJIT_FLAGS, unsigned int*) + 0xa5e [/__w/1/s/src/coreclr/vm/jitinterface.cpp:0], calling libcoreclr.so!invokeCompileMethod(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12430]
00007F89A080C320 00007f8a1b009a4d libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x27d [/__w/1/s/src/coreclr/utilcode/util.cpp:0], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C340 00007f8a1b0c9d95 libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) + 0x15 [/__w/1/s/src/coreclr/md/inc/metamodel.h:1438]
00007F89A080C3A0 00007f8a1b0c9d95 libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) + 0x15 [/__w/1/s/src/coreclr/md/inc/metamodel.h:1438]
00007F89A080C3C0 00007f8a1b0cc3a1 libcoreclr.so!MDInternalRO::GetCountWithTokenKind(unsigned int) + 0x11 [/__w/1/s/src/coreclr/md/runtime/mdinternalro.cpp:175], calling libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) [/__w/1/s/src/coreclr/md/inc/metamodel.h:1437]
00007F89A080C3E0 00007f8a1b009968 libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x198 [/__w/1/s/src/coreclr/utilcode/util.cpp:0]
00007F89A080C440 00007f8a1b009d5b libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x58b [/__w/1/s/src/coreclr/utilcode/util.cpp:1859], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C4A0 00007f8a1b00a32b libcoreclr.so!validateTokenSig(unsigned int, unsigned char const*, unsigned int, unsigned int, IMDInternalImport*) + 0x2eb [/__w/1/s/src/coreclr/utilcode/util.cpp:1972], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C620 00007f8a1acd5e8a libcoreclr.so!MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*, unsigned int*, CORJIT_FLAGS*) + 0x2ea [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, CORJIT_FLAGS, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12717]
00007F89A080C650 00007f8a1ac1728b libcoreclr.so!MethodDesc::GetPrecode() + 0x10b [/__w/1/s/src/coreclr/inc/check.inl:22], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080C750 00007f8a1acd59b6 libcoreclr.so!MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) + 0x2c6 [/__w/1/s/src/coreclr/vm/prestub.cpp:814], calling libcoreclr.so!MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*, unsigned int*, CORJIT_FLAGS*) [/__w/1/s/src/coreclr/vm/prestub.cpp:925]
00007F89A080C7B0 00007f8a1ad1606c libcoreclr.so!DeadlockAwareLock::EndEnterLock() + 0x17c [/__w/1/s/src/coreclr/inc/check.inl:22], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080C830 00007f8a1acd4f7d libcoreclr.so!MethodDesc::JitCompileCode(PrepareCodeConfig*) + 0x3ed [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) [/__w/1/s/src/coreclr/vm/prestub.cpp:760]
00007F89A080C900 00007f8a1acd4368 libcoreclr.so!MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 0x1f8 [/__w/1/s/src/coreclr/vm/prestub.cpp:426], calling libcoreclr.so!MethodDesc::JitCompileCode(PrepareCodeConfig*) [/__w/1/s/src/coreclr/vm/prestub.cpp:640]
00007F89A080C910 00007f8a1abdc250 libcoreclr.so!MethodDesc::GetCustomAttribute(WellKnownAttribute, void const**, unsigned int*) const + 0x240 [/__w/1/s/src/coreclr/vm/ceeload.h:0]
00007F89A080C950 00007f8a1acd6d32 libcoreclr.so!PrepareCodeConfigBuffer::PrepareCodeConfigBuffer(NativeCodeVersion) + 0x102 [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!VersionedPrepareCodeConfig::FinishConfiguration() [/__w/1/s/src/coreclr/vm/prestub.cpp:1323]
00007F89A080C9B0 00007f8a1ad1ba0a libcoreclr.so!TieredCompilationManager::CompileCodeVersion(NativeCodeVersion) + 0x1fa [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!MethodDesc::PrepareCode(PrepareCodeConfig*) [/__w/1/s/src/coreclr/vm/prestub.cpp:317]
00007F89A080CAB0 00007f8a1ad1a863 libcoreclr.so!TieredCompilationManager::DoBackgroundWork(unsigned long*, unsigned long, unsigned long) + 0x3f3 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:877], calling libcoreclr.so!TieredCompilationManager::CompileCodeVersion(NativeCodeVersion) [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:886]
00007F89A080CB50 00007f8a1b21268b libcoreclr.so!CorUnix::InternalSleepEx(CorUnix::CPalThread*, unsigned int, int) + 0x16b [/__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:888], calling libcoreclr.so!___lldb_unnamed_symbol30612
00007F89A080CBD0 00007f8a1ad19948 libcoreclr.so!TieredCompilationManager::BackgroundWorkerStart() + 0x178 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:482], calling libcoreclr.so!TieredCompilationManager::DoBackgroundWork(unsigned long*, unsigned long, unsigned long) [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:661]
00007F89A080CC30 00007f8a1ad19793 libcoreclr.so!TieredCompilationManager::BackgroundWorkerBootstrapper1(void*) + 0x33 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!TieredCompilationManager::BackgroundWorkerStart() [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:435]
00007F89A080CC60 00007f8a1ad13c8e libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x24e [/__w/1/s/src/coreclr/vm/threads.cpp:0]
00007F89A080CC80 00007f8a1abfc918 libcoreclr.so!CrstBase::Leave() + 0x98 [/__w/1/s/src/coreclr/vm/util.hpp:498], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080CD00 00007f8a1aeeac15 libcoreclr.so!ETW::ThreadLog::FireThreadCreated(Thread*) + 0xf5 [/__w/1/s/artifacts/obj/coreclr/Linux.x64.Checked/inc/clretwallmain.h:1658], calling libcoreclr.so!EventPipeWriteEventThreadCreated [/__w/1/s/artifacts/obj/coreclr/Linux.x64.Checked/vm/eventing/eventpipe/eventpipe/dotnetruntime.cpp:4553]
00007F89A080CD40 00007f8a1aba653c libcoreclr.so!CAutoTryCleanup<CLRException::HandlerState>::~CAutoTryCleanup() + 0xc [/__w/1/s/src/coreclr/inc/ex.h:616], calling libcoreclr.so!CLRException::HandlerState::CleanupTry() [/__w/1/s/src/coreclr/vm/clrex.cpp:791]
00007F89A080CD50 00007f8a1ad08769 libcoreclr.so!Thread::HasStarted() + 0x3b9 [/__w/1/s/src/coreclr/vm/threads.cpp:1911], calling libcoreclr.so!LoggingEnabled() [/__w/1/s/src/coreclr/utilcode/log.cpp:234]
00007F89A080CDC0 00007f8a1ad1445d libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) + 0x2d [/__w/1/s/src/coreclr/vm/threads.cpp:0], calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [/__w/1/s/src/coreclr/vm/threads.cpp:7457]
00007F89A080CDF0 00007f8a1ad196f0 libcoreclr.so!TieredCompilationManager::BackgroundWorkerBootstrapper0(void*) + 0xc0 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) [/__w/1/s/src/coreclr/vm/threads.cpp:7583]
00007F89A080CE30 00007f8a1b21eca2 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x3c2 [/__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1829]
00007F89A080CEF0 00007f8a1c24b6db 00007f8a1c24b6db
00007F89A080CFB0 00007f8a1b43161f 00007f8a1b43161f

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We have a few crashes on the runtime-coreclr libraries-pgo leg on Ubuntu 18.04, both ARM and x64.

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9c87c5c4deb0499081/System.Net.Http.Unit.Tests/1/console.5bbac1e3.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d9d770df61c8470993/System.Net.Http.Unit.Tests/1/console.a9395c58.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-62ca6e33b69446b7b9/System.Net.Http.Unit.Tests/1/console.78831c68.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-07b013262b854ee2b3/System.Net.Http.Unit.Tests/1/console.c435ec1c.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9d085c25e76846d882/System.Net.Http.Unit.Tests/1/console.ceac850a.log?helixlogtype=result
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d09ca73c69e646cfbe/System.Net.Http.Unit.Tests/1/console.c9863e43.log?helixlogtype=result

I checked the crash dump for the first one:

lldb dumpstack

(lldb) dumpstack
OS Thread Id: 0x14d7 (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libclrjit.so!Compiler::optMarkLoopRemoved(unsigned int) + 0xdd [/__w/1/s/src/coreclr/jit/optimizer.cpp:10093]
Child-SP         RetAddr          Caller, Callee
00007F89A080AC30 00007f89a0b51ea6 libclrjit.so!Compiler::optUpdateLoopsBeforeRemoveBlock(BasicBlock*, bool) + 0x206 [/__w/1/s/src/coreclr/jit/optimizer.cpp:437], calling libclrjit.so!Compiler::optMarkLoopRemoved(unsigned int) [/__w/1/s/src/coreclr/jit/optimizer.cpp:10085]
00007F89A080ACB0 00007f89a09e155e libclrjit.so!Compiler::fgUnreachableBlock(BasicBlock*) + 0x1ae [/__w/1/s/src/coreclr/jit/block.h:605], calling libclrjit.so!Compiler::optUpdateLoopsBeforeRemoveBlock(BasicBlock*, bool) [/__w/1/s/src/coreclr/jit/optimizer.cpp:394]
00007F89A080ACE0 00007f89a09c1f21 libclrjit.so!Compiler::fgRemoveBlock(BasicBlock*, bool) + 0x291 [/__w/1/s/src/coreclr/jit/fgbasic.cpp:4573], calling libclrjit.so!Compiler::fgUnreachableBlock(BasicBlock*) [/__w/1/s/src/coreclr/jit/fgopt.cpp:2423]
00007F89A080AD80 00007f89a09e6ab7 libclrjit.so!Compiler::fgUpdateFlowGraph(bool) + 0x9a7 [/__w/1/s/src/coreclr/jit/fgopt.cpp:0], calling libclrjit.so!Compiler::fgRemoveBlock(BasicBlock*, bool) [/__w/1/s/src/coreclr/jit/fgbasic.cpp:4540]
00007F89A080ADF0 00007f89a0b5acd4 libclrjit.so!Compiler::optOptimizeLayout() + 0x94 [/__w/1/s/src/coreclr/jit/optimizer.cpp:5164], calling libclrjit.so!Compiler::fgUpdateFlowGraph(bool) [/__w/1/s/src/coreclr/jit/fgopt.cpp:5876]
00007F89A080AE10 00007f89a0b6ada6 libclrjit.so!Phase::Run() + 0x76 [/__w/1/s/src/coreclr/jit/phase.cpp:61]
00007F89A080AE60 00007f89a0994edd libclrjit.so!Compiler::compCompile(void**, unsigned int*, JitFlags*) + 0x139d [/__w/1/s/src/coreclr/jit/phase.h:37], calling libclrjit.so!Phase::Run() [/__w/1/s/src/coreclr/jit/phase.cpp:58]
00007F89A080AF60 00007f89a0a9eadf libclrjit.so!Compiler::fgVerifyHandlerTab() + 0xf0f [/__w/1/s/src/coreclr/jit/jiteh.cpp:0], calling libclrjit.so!___lldb_unnamed_symbol8237
00007F89A080AFA0 00007f8a1ac7822c libcoreclr.so!CEEInfo::logMsg(unsigned int, char const*, __va_list_tag*) + 0x1c [/__w/1/s/src/coreclr/vm/jitinterface.cpp:10703], calling libcoreclr.so!LoggingOn(unsigned int, unsigned int) [/__w/1/s/src/coreclr/utilcode/log.cpp:238]
00007F89A080B120 00007f89a0998f01 libclrjit.so!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) + 0xdb1 [/__w/1/s/src/coreclr/jit/compiler.hpp:3877], calling libclrjit.so!Compiler::compCompile(void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:4325]
00007F89A080B1D0 00007f89a0997277 libclrjit.so!Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*) + 0x8a7 [/__w/1/s/src/coreclr/jit/compiler.cpp:0], calling libclrjit.so!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:6292]
00007F89A080B200 00007f89a0971bd3 libclrjit.so!CodeGen::CodeGen(Compiler*) + 0x193 [/__w/1/s/src/coreclr/jit/jit.h:782], calling libclrjit.so!Compiler::compGetJitDefaultFill(Compiler*) [/__w/1/s/src/coreclr/jit/compiler.cpp:2054]
00007F89A080B240 00007f89a09719c3 libclrjit.so!getCodeGenerator(Compiler*) + 0x53 [/__w/1/s/src/coreclr/jit/codegencommon.cpp:62], calling libclrjit.so!CodeGen::CodeGen(Compiler*) [/__w/1/s/src/coreclr/jit/codegencommon.cpp:74]
00007F89A080B260 00007f89a098fafd libclrjit.so!Compiler::compInit(ArenaAllocator*, CORINFO_METHOD_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, InlineInfo*) + 0x4ed [/__w/1/s/src/coreclr/jit/jitstd/list.h:301], calling libclrjit.so!ArenaAllocator::getMemStatsAllocator(CompMemKind) [/__w/1/s/src/coreclr/jit/alloc.cpp:283]
00007F89A080B2B0 00007f89a0999e61 libclrjit.so!jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) + 0x341 [/__w/1/s/src/coreclr/jit/compiler.cpp:7375], calling libclrjit.so!Compiler::compCompile(CORINFO_MODULE_STRUCT_*, void**, unsigned int*, JitFlags*) [/__w/1/s/src/coreclr/jit/compiler.cpp:5475]
00007F89A080B3B0 00007f8a1ad28a67 libcoreclr.so!EETypeHashTable::CompareInstantiatedType(TypeHandle, Module*, unsigned int, Instantiation) + 0xd7 [/__w/1/s/src/coreclr/vm/typehandle.h:698], calling libcoreclr.so!TypeHandle::GetInstantiation() const [/__w/1/s/src/coreclr/vm/typehandle.cpp:361]
00007F89A080B8D0 00007f89a09a65c6 libclrjit.so!CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) + 0x116 [/__w/1/s/src/coreclr/jit/ee_il_dll.cpp:276], calling libclrjit.so!jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) [/__w/1/s/src/coreclr/jit/compiler.cpp:7261]
00007F89A080B970 00007f8a1ac7cede libcoreclr.so!invokeCompileMethodHelper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) + 0x1ae [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12379]
00007F89A080B980 00007f8a1aba656a libcoreclr.so!Thread::EnablePreemptiveGC() + 0x1a [/__w/1/s/src/coreclr/vm/threads.inl:42], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080B9C0 00007f8a1ac7cf7a libcoreclr.so!invokeCompileMethod(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) + 0x5a [/__w/1/s/src/coreclr/vm/jitinterface.cpp:0], calling libcoreclr.so!invokeCompileMethodHelper(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12284]
00007F89A080BA30 00007f8a1ac7e02e libcoreclr.so!UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, CORJIT_FLAGS, unsigned int*) + 0xa5e [/__w/1/s/src/coreclr/vm/jitinterface.cpp:0], calling libcoreclr.so!invokeCompileMethod(EEJitManager*, CEEInfo*, CORINFO_METHOD_INFO*, CORJIT_FLAGS, unsigned char**, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12430]
00007F89A080C320 00007f8a1b009a4d libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x27d [/__w/1/s/src/coreclr/utilcode/util.cpp:0], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C340 00007f8a1b0c9d95 libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) + 0x15 [/__w/1/s/src/coreclr/md/inc/metamodel.h:1438]
00007F89A080C3A0 00007f8a1b0c9d95 libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) + 0x15 [/__w/1/s/src/coreclr/md/inc/metamodel.h:1438]
00007F89A080C3C0 00007f8a1b0cc3a1 libcoreclr.so!MDInternalRO::GetCountWithTokenKind(unsigned int) + 0x11 [/__w/1/s/src/coreclr/md/runtime/mdinternalro.cpp:175], calling libcoreclr.so!CMiniMdTemplate<CMiniMd>::CommonGetRowCount(unsigned int) [/__w/1/s/src/coreclr/md/inc/metamodel.h:1437]
00007F89A080C3E0 00007f8a1b009968 libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x198 [/__w/1/s/src/coreclr/utilcode/util.cpp:0]
00007F89A080C440 00007f8a1b009d5b libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) + 0x58b [/__w/1/s/src/coreclr/utilcode/util.cpp:1859], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C4A0 00007f8a1b00a32b libcoreclr.so!validateTokenSig(unsigned int, unsigned char const*, unsigned int, unsigned int, IMDInternalImport*) + 0x2eb [/__w/1/s/src/coreclr/utilcode/util.cpp:1972], calling libcoreclr.so!validateOneArg(unsigned int, SigParser*, unsigned int*, IMDInternalImport*, int) [/__w/1/s/src/coreclr/utilcode/util.cpp:1659]
00007F89A080C620 00007f8a1acd5e8a libcoreclr.so!MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*, unsigned int*, CORJIT_FLAGS*) + 0x2ea [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!UnsafeJitFunction(PrepareCodeConfig*, COR_ILMETHOD_DECODER*, CORJIT_FLAGS, unsigned int*) [/__w/1/s/src/coreclr/vm/jitinterface.cpp:12717]
00007F89A080C650 00007f8a1ac1728b libcoreclr.so!MethodDesc::GetPrecode() + 0x10b [/__w/1/s/src/coreclr/inc/check.inl:22], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080C750 00007f8a1acd59b6 libcoreclr.so!MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) + 0x2c6 [/__w/1/s/src/coreclr/vm/prestub.cpp:814], calling libcoreclr.so!MethodDesc::JitCompileCodeLocked(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*, unsigned int*, CORJIT_FLAGS*) [/__w/1/s/src/coreclr/vm/prestub.cpp:925]
00007F89A080C7B0 00007f8a1ad1606c libcoreclr.so!DeadlockAwareLock::EndEnterLock() + 0x17c [/__w/1/s/src/coreclr/inc/check.inl:22], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080C830 00007f8a1acd4f7d libcoreclr.so!MethodDesc::JitCompileCode(PrepareCodeConfig*) + 0x3ed [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) [/__w/1/s/src/coreclr/vm/prestub.cpp:760]
00007F89A080C900 00007f8a1acd4368 libcoreclr.so!MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 0x1f8 [/__w/1/s/src/coreclr/vm/prestub.cpp:426], calling libcoreclr.so!MethodDesc::JitCompileCode(PrepareCodeConfig*) [/__w/1/s/src/coreclr/vm/prestub.cpp:640]
00007F89A080C910 00007f8a1abdc250 libcoreclr.so!MethodDesc::GetCustomAttribute(WellKnownAttribute, void const**, unsigned int*) const + 0x240 [/__w/1/s/src/coreclr/vm/ceeload.h:0]
00007F89A080C950 00007f8a1acd6d32 libcoreclr.so!PrepareCodeConfigBuffer::PrepareCodeConfigBuffer(NativeCodeVersion) + 0x102 [/__w/1/s/src/coreclr/vm/prestub.cpp:0], calling libcoreclr.so!VersionedPrepareCodeConfig::FinishConfiguration() [/__w/1/s/src/coreclr/vm/prestub.cpp:1323]
00007F89A080C9B0 00007f8a1ad1ba0a libcoreclr.so!TieredCompilationManager::CompileCodeVersion(NativeCodeVersion) + 0x1fa [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!MethodDesc::PrepareCode(PrepareCodeConfig*) [/__w/1/s/src/coreclr/vm/prestub.cpp:317]
00007F89A080CAB0 00007f8a1ad1a863 libcoreclr.so!TieredCompilationManager::DoBackgroundWork(unsigned long*, unsigned long, unsigned long) + 0x3f3 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:877], calling libcoreclr.so!TieredCompilationManager::CompileCodeVersion(NativeCodeVersion) [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:886]
00007F89A080CB50 00007f8a1b21268b libcoreclr.so!CorUnix::InternalSleepEx(CorUnix::CPalThread*, unsigned int, int) + 0x16b [/__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:888], calling libcoreclr.so!___lldb_unnamed_symbol30612
00007F89A080CBD0 00007f8a1ad19948 libcoreclr.so!TieredCompilationManager::BackgroundWorkerStart() + 0x178 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:482], calling libcoreclr.so!TieredCompilationManager::DoBackgroundWork(unsigned long*, unsigned long, unsigned long) [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:661]
00007F89A080CC30 00007f8a1ad19793 libcoreclr.so!TieredCompilationManager::BackgroundWorkerBootstrapper1(void*) + 0x33 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!TieredCompilationManager::BackgroundWorkerStart() [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:435]
00007F89A080CC60 00007f8a1ad13c8e libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x24e [/__w/1/s/src/coreclr/vm/threads.cpp:0]
00007F89A080CC80 00007f8a1abfc918 libcoreclr.so!CrstBase::Leave() + 0x98 [/__w/1/s/src/coreclr/vm/util.hpp:498], calling libcoreclr.so!___lldb_unnamed_symbol30368
00007F89A080CD00 00007f8a1aeeac15 libcoreclr.so!ETW::ThreadLog::FireThreadCreated(Thread*) + 0xf5 [/__w/1/s/artifacts/obj/coreclr/Linux.x64.Checked/inc/clretwallmain.h:1658], calling libcoreclr.so!EventPipeWriteEventThreadCreated [/__w/1/s/artifacts/obj/coreclr/Linux.x64.Checked/vm/eventing/eventpipe/eventpipe/dotnetruntime.cpp:4553]
00007F89A080CD40 00007f8a1aba653c libcoreclr.so!CAutoTryCleanup<CLRException::HandlerState>::~CAutoTryCleanup() + 0xc [/__w/1/s/src/coreclr/inc/ex.h:616], calling libcoreclr.so!CLRException::HandlerState::CleanupTry() [/__w/1/s/src/coreclr/vm/clrex.cpp:791]
00007F89A080CD50 00007f8a1ad08769 libcoreclr.so!Thread::HasStarted() + 0x3b9 [/__w/1/s/src/coreclr/vm/threads.cpp:1911], calling libcoreclr.so!LoggingEnabled() [/__w/1/s/src/coreclr/utilcode/log.cpp:234]
00007F89A080CDC0 00007f8a1ad1445d libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) + 0x2d [/__w/1/s/src/coreclr/vm/threads.cpp:0], calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [/__w/1/s/src/coreclr/vm/threads.cpp:7457]
00007F89A080CDF0 00007f8a1ad196f0 libcoreclr.so!TieredCompilationManager::BackgroundWorkerBootstrapper0(void*) + 0xc0 [/__w/1/s/src/coreclr/vm/tieredcompilation.cpp:0], calling libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) [/__w/1/s/src/coreclr/vm/threads.cpp:7583]
00007F89A080CE30 00007f8a1b21eca2 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x3c2 [/__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1829]
00007F89A080CEF0 00007f8a1c24b6db 00007f8a1c24b6db
00007F89A080CFB0 00007f8a1b43161f 00007f8a1b43161f

Author:	rzikm
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[AndyAyersMS]

Another case where loop blocks get reordered, so that walking from top->bottom via bbnext does not work out as expected.

Here lpTop == BB24, lpBottom = BB31.

BB31  [0079]  0  0                          0          0  2 [???..???)-> BB24  ( cond ) T0                  internal rare bwd IBC 
BB198 [0353]  2  0    BB28,BB31             0             2 [???..???)-> BB32  (always) T0                  internal rare 
BB28  [0077]  1  0    BB27                  0          0  2 [000..000)-> BB198 ( cond ) T0                  i internal rare bwd IBC 
BB24  [0052]  2  0    BB28,BB31             0          0  2 [000..000)-> BB25  (always) T0                  i internal rare Loop bwd bwd-target IBC align 

and so optUnmarkLoopBlocks walks off the end of the bblist.

This reordering happens quite a bit earlier; here '[0052]is the top and[0079]` is the bottom block. Initially they're all in a row:

BB24  [0052]  1  0    BB31                  0          0  2 [000..000)                  T0                  i internal rare Loop bwd bwd-target IBC align 
BB25  [0053]  2  0    BB21,BB24            23.35     864  2 [000..000)-> BB32  ( cond ) T0                  i internal bwd IBC 
BB26  [0054]  1  0    BB25                 23.29     862  2 [000..000)-> BB32  ( cond ) T0                  i internal idxlen bwd IBC 
BB27  [0055]  1  0    BB26                 19.64     727  2 [000..000)-> BB29  ( cond ) T0                  i internal idxlen bwd bwd-src IBC 
BB28  [0077]  1  0    BB27                  0          0  2 [000..000)-> BB31  (always) T0                  i internal rare bwd IBC 
BB29  [0078]  1  0    BB27                314.29   11629  2 [000..000)-> BB32  (always) T0                  i internal bwd IBC 
BB31  [0079]  1  0    BB28                  0          0  2 [???..???)-> BB24  ( cond ) T0                  internal rare bwd IBC 

but then some random blocks are inserted before BB28, and then this moves BB24 down below other blocks in the loop

Decided to straighten unconditional branch at block BB21 branch to BB25 because of IBC profile data
fgFindInsertPoint(regionIndex=1, putInTryRegion=true, startBlk=BB02, endBlk=BB188, nearBlk=BB25, jumpBlk=BB00, runRarely=true)
Relocated rarely run block BB24
Relocated block [BB24..BB24] inserted after BB28
Changed an unconditional jump from BB21 to the next block BB25 into a BBJ_NONE block
Block BB24 ended with a BBJ_NONE, Changed to an unconditional jump to BB25

BB25  [0053]  2  0    BB21,BB24            23.35     864  2 [000..000)-> BB32  ( cond ) T0                  i internal bwd IBC 
BB26  [0054]  1  0    BB25                 23.29     862  2 [000..000)-> BB32  ( cond ) T0                  i internal idxlen bwd IBC 
BB27  [0055]  1  0    BB26                 19.64     727  2 [000..000)-> BB29  ( cond ) T0                  i internal idxlen bwd bwd-src IBC 
BB28  [0077]  1  0    BB27                  0          0  2 [000..000)-> BB31  (always) T0                  i internal rare bwd IBC 
BB24  [0052]  1  0    BB31                  0          0  2 [000..000)-> BB25  (always) T0                  i internal rare Loop bwd bwd-target IBC align 
BB196 [0351]  1  0    BB133                 0             0 [???..???)-> BB134 (always) T0                  internal rare 
BB192 [0347]  1  0    BB115                 0             3 [???..???)-> BB116 (always) T0                  internal rare 
BB17  [0073]  1  0    BB191                 0          0  1 [000..000)                  T0                  i internal rare bwd IBC 
BB20  [0075]  1  0    BB17                  0          0  1 [???..???)-> BB14  ( cond ) T0                  internal rare bwd IBC 
BB197 [0352]  1  0    BB20                  0             1 [???..???)-> BB21  (always) T0                  internal rare 
BB29  [0078]  1  0    BB27                314.29   11629  2 [000..000)-> BB32  (always) T0                  i internal bwd IBC 
BB31  [0079]  1  0    BB28                  0          0  2 [???..???)-> BB24  ( cond ) T0                  internal rare bwd IBC 

and so on.

While it would be nice to fix this sort of reordering, downstream code can't expect a loop body to remain contiguous in a bb chain, so needs to be prepared.

Seems like the only fix is to prewalk and ensure we find bottom from top before we hit the end, and if not, give up on unmarking? Or walk all blocks so the order is immaterial?

[AndyAyersMS]

Seems like one issue is that fgReorderBlocks can relocate rarely run blocks and/or ranges it in a way that tends to scramble their order.

And if a loop entry is rarely run but some loop blocks are not (as in the example above) this can end up both moving top and bottom away from the rest of the loop nest and also reversing their order.

[AndyAyersMS]

Maybe the issue is that while the random profile data is self-consistent at the flow graph level, it may not be consistent with actual flows allowed by the IR. Say we have a redundant branch in X and Y. Random profile might end up with this self-consistent but nonsensical set of weights:

flowchart TD
  X["X \n100\n if (p)"] --> A["A\n0"] & B["B\n100"]
  A & B --> Y["Y\n100\n if (p)"]
  Y --> C["C\n100"] & D["D\n0"]
  C & D --> Z["Z\n100"]

Loading

and post RBO we would end up with inconsistent data, like seeing that a loop entry has zero weight but loop body blocks have nonzero weight.

At a fundamental level inconsistent profile data during compilation is inevitable. When we inline or do other block duplication we have to make somewhat arbitrary choices about how the profile data applies to the newly created blocks, and we will (frequently) get it wrong. But we're not well set up currently to handle it properly.