Internal CLR errors during delay load when using crossgen2 in composite mode with method layout

[jakobbotsch]

When passing --method-layout and --composite to crossgen2 I get internal CLR errors when running the produced assemblies. Here are some steps to reproduce it:

1. Apply the following patch so we don't need to use PGO data to do method layout:

commit eed090664adb6aae53385a3cb8a07649741b9538
Author: Jakob Botsch Nielsen <jakob.botsch.nielsen@gmail.com>
Date:   Mon Sep 13 20:49:14 2021 +0200

    WIP

diff --git a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunFileLayoutOptimizer.cs b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunFileLayoutOptimizer.cs
index d8d6d36191a..00ebe064e9d 100644
--- a/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunFileLayoutOptimizer.cs
+++ b/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunFileLayoutOptimizer.cs
@@ -26,6 +26,7 @@ public enum ReadyToRunMethodLayoutAlgorithm
         HotWarmCold,
         CallFrequency,
         PettisHansen,
+        Random,
     }
 
     public enum ReadyToRunFileLayoutAlgorithm
@@ -166,6 +167,18 @@ int ComputeHotWarmColdRegion(MethodWithGCInfo method)
                     methods = PettisHansenSort(methods);
                     break;
 
+                case ReadyToRunMethodLayoutAlgorithm.Random:
+                    Random rand = new Random(0);
+                    for (int i = 0; i < methods.Count - 1; i++)
+                    {
+                        int j = rand.Next(i, methods.Count);
+                        MethodWithGCInfo temp = methods[i];
+                        methods[i] = methods[j];
+                        methods[j] = temp;
+                    }
+
+                    break;
+
                 default:
                     throw new NotImplementedException(_methodLayoutAlgorithm.ToString());
             }
diff --git a/src/coreclr/tools/aot/crossgen2/Program.cs b/src/coreclr/tools/aot/crossgen2/Program.cs
index 2d810c9c799..f25fa36e5c8 100644
--- a/src/coreclr/tools/aot/crossgen2/Program.cs
+++ b/src/coreclr/tools/aot/crossgen2/Program.cs
@@ -156,6 +156,7 @@ private void ProcessCommandLine(string[] args)
                     "hotwarmcold" => ReadyToRunMethodLayoutAlgorithm.HotWarmCold,
                     "callfrequency" => ReadyToRunMethodLayoutAlgorithm.CallFrequency,
                     "pettishansen" => ReadyToRunMethodLayoutAlgorithm.PettisHansen,
+                    "random" => ReadyToRunMethodLayoutAlgorithm.Random,
                     _ => throw new CommandLineException(SR.InvalidMethodLayout)
                 };
             }

2. Create a simple .NET 6 console app with the following code:

using System;
using System.Collections.Generic;

public class Program
{
	public static void Main()
	{
		Dictionary<string, string> test = new Dictionary<string, string>(StringComparer.Ordinal);
		test["foo"] = "bar";
	}
}

3. Publish the app self-contained, copy debug bits and invoke crossgen2 with --composite and --method-layout:random:

dotnet publish -c Release --self-contained -r win-x64 -o publish
cp ..\dotnet\runtime\artifacts\bin\coreclr\windows.x64.Debug\coreclr.dll .\publish\
cp ..\dotnet\runtime\artifacts\bin\coreclr\windows.x64.Debug\clrjit.dll .\publish\
cp ..\dotnet\runtime\artifacts\bin\coreclr\windows.x64.Debug\System.Private.CoreLib.dll .\publish\
cd publish
mkdir r2r
..\..\dotnet\runtime\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\crossgen2\crossgen2.exe --targetos:windows --targetarch:x64 -O --composite --method-layout:random --out:r2r\repro.R2R.dll *.dll
cp r2r\* .
.\repro.exe

On my system I get:

Assert failure(PID 34376 [0x00008648], Thread: 25384 [0x6328]): Consistency check failed: AV in clr at this callstack:
------
CORECLR! ReadyToRunJitManager::JitCodeToMethodInfo + 0x32D (0x00007ffd`6f1d546d)
CORECLR! EECodeInfo::Init + 0x225 (0x00007ffd`6e69bb85)
CORECLR! EECodeInfo::Init + 0x18A (0x00007ffd`6e69b93a)
CORECLR! EECodeInfo::EECodeInfo + 0x2F (0x00007ffd`6e5d4b6f)
CORECLR! ExecutionManager::GetCodeMethodDesc + 0x194 (0x00007ffd`6f1cf164)
CORECLR! MethodTable::GetMethodDescForSlotAddress + 0x3EF (0x00007ffd`6e7415ff)
CORECLR! DispatchSlot::GetMethodDesc + 0x42 (0x00007ffd`6e594c22)
CORECLR! VirtualCallStubManager::Resolver + 0x54E (0x00007ffd`6e88f67e)
CORECLR! VirtualCallStubManager::ResolveWorker + 0xB29 (0x00007ffd`6e88e619)
CORECLR! VSD_ResolveWorker + 0xACB (0x00007ffd`6e893e3b)
CORECLR! ResolveWorkerAsmStub + 0x6D (0x00007ffd`6eecf51d)
REPRO.R2R! <no symbol> + 0x0 (0x00007ffd`570503f7)
<no module>! <no symbol> + 0x0 (0x00000238`b5631628)
<no module>! <no symbol> + 0x0 (0x00000238`b5631aa8)
<no module>! <no symbol> + 0x0 (0x00000238`b5631240)
<no module>! <no symbol> + 0x0 (0x00000000`00000002)
<no module>! <no symbol> + 0x0 (0x00000238`b344e520)
CORECLR! DelayLoad_MethodCall + 0x71 (0x00007ffd`6eecf111)
REPRO.R2R! <no symbol> + 0x0 (0x00007ffd`580af895)
-----
.AV on tid=0x6328 (25384), cxr=000000B6E677AE80, exr=000000B6E677B370
FAILED: false

CORECLR! CHECK::Trigger + 0x299 (0x00007ffd`6eeff6c9)
CORECLR! CLRVectoredExceptionHandlerPhase3 + 0x374 (0x00007ffd`6e5f8f44)
CORECLR! CLRVectoredExceptionHandlerPhase2 + 0xA6 (0x00007ffd`6e5f8836)
CORECLR! CLRVectoredExceptionHandler + 0x56A (0x00007ffd`6e5f874a)
CORECLR! CLRVectoredExceptionHandlerShim + 0x2CD (0x00007ffd`6e5f938d)
NTDLL! RtlDeleteAce + 0x3CC (0x00007ffd`fb288a4c)
NTDLL! RtlRaiseException + 0x256 (0x00007ffd`fb261276)
NTDLL! KiUserExceptionDispatcher + 0x2E (0x00007ffd`fb2b0cae)
CORECLR! ReadyToRunJitManager::JitCodeToMethodInfo + 0x32D (0x00007ffd`6f1d546d)
CORECLR! EECodeInfo::Init + 0x225 (0x00007ffd`6e69bb85)
    File: C:\dev\dotnet\runtime\src\coreclr\vm\excep.cpp Line: 7581
    Image: C:\dev\dotnet\investigations\methodorder_TE\repro2\publish\repro.exe

cc @dotnet/crossgen-contrib

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[davidwrighton]

Oh, that's a great idea for testing. We should make that randomization mode part of one of our R2R tests so that once we fix this it doesn't come back.