Many tests failed with Verify_FieldOffset 'ExtendedSocketException._endPoint' Field offset 128!=136(actual) || baseOffset 128!=136(actual)

[VincentBu]

Run: runtime-coreclr crossgen2 20210320.1

One of error message:

/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.sh: line 280: -r:/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/IL-CG2/.dll: No such file or directory
rm: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/common.dll.rsp: No such file or directory
rm: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.dll.rsp: No such file or directory

Assert failure(PID 1344 [0x00000540], Thread: 1065195 [0x1040eb]): Verify_FieldOffset 'ExtendedSocketException._endPoint' Field offset 128!=136(actual) || baseOffset 128!=136(actual)
 File: /Users/runner/work/1/s/src/coreclr/vm/jitinterface.cpp Line: 14129
 Image: /private/tmp/helix/working/AD85094A/p/corerun

MachO coredumps are not supported. To enable ELF coredumps on MacOS, set the COMPlus_DbgEnableElfDumpOnMacOS environment variable to 1.
/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.sh: line 401: 1344 Abort trap: 6 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code: 1
Raw output file: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/Reports/tracing.eventpipe/processinfo/processinfo/processinfo.output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/IL-CG2/common.dll
Response file: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/common.dll.rsp
/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/IL-CG2/common.dll
-o:/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/common.dll
-r:/tmp/helix/working/AD85094A/p/System..dll
-r:/tmp/helix/working/AD85094A/p/Microsoft..dll
-r:/tmp/helix/working/AD85094A/p/mscorlib.dll
--verify-type-and-field-layout
--targetarch:x64
-O
Running CrossGen2: dotnet /tmp/helix/working/AD85094A/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/common.dll.rsp 
Emitting R2R PE file: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/common.dll
/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/IL-CG2/processinfo.dll
Response file: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.dll.rsp
/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/IL-CG2/processinfo.dll
-o:/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.dll
-r:/tmp/helix/working/AD85094A/p/System..dll
-r:/tmp/helix/working/AD85094A/p/Microsoft.*.dll
-r:/tmp/helix/working/AD85094A/p/mscorlib.dll
--verify-type-and-field-layout
--targetarch:x64
-O
Running CrossGen2: dotnet /tmp/helix/working/AD85094A/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.dll.rsp 
Emitting R2R PE file: /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.dll
in ReleaseLock
/tmp/helix/working/AD85094A/p/corerun processinfo.dll ''
 0.0s: Test PID: 1344
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/AD85094A/p
> /private/tmp/helix/working/AD85094A/w/B37709B8/e/tracing/eventpipe/processinfo/processinfo/processinfo.sh
Expected: True
Actual: False

Stack trace
   at tracing_eventpipe._processinfo_processinfo_processinfo_._processinfo_processinfo_processinfo_sh()

[mangod9]

cc @trylek @dotnet/crossgen-contrib. Possibly due to optimizations which were recently enabled?

[davidwrighton]

This is likely caused by @trylek's change to make our cg2 testing more similar to how we are shipping preview3. Looks to be a small bug in the type layout engine of crossgen2.

[davidwrighton]

(Note that this isn't likely newly introduced, just newly found)

[trylek]

I finally understand the underlying problem and I believe it exposes a pre-existing debt from the time of composite implementation we need to figure out how to address: the discrepancy is basically between these two lines of code:

runtime/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCompilationModuleGroupBase.cs

Line 272 in 5533388

if (!ModuleMatchesCompilationUnitIndex(derivedType.Module, baseType.Module) ||

and

runtime/src/coreclr/vm/methodtablebuilder.cpp

Line 11059 in 5533388

return module1->GetCompositeNativeImage() == module2->GetCompositeNativeImage();

The Crossgen2 version basically states that anything outside of the current version bubble is "some other big version bubble" which is true for composite framework but not for single-file framework with large bubble turned off. To support the fully general case we'd probably need to have a way of specifying version bubble boundaries for reference assemblies in Crossgen2. At the very least we'd probably need a boolean flag stating whether the outside of the current version bubble is one bubble or not but that won't support hierarchical version bubbles (e.g. ASP.NET composite on top of framework composite).

I actually think it shouldn't be rocket science to implement even the general case by using a state machine - perhaps a special option, something like --bubble-boundary, that would be used as a reference assembly separator declaring the version bubble boundaries; Alternatively we may be able to devise some extended syntax for reference assemblies that would allow specifying some version bubble ID for each reference - something like -r[FW]:System.*.dll, -r[ASP]:Microsoft.AspNet.*.dll, I just made these up purely for demonstrational purposes.

While the implementation on the Crossgen2 side is mostly trivial in all the proposed cases, I'm reluctant to dive into it before I hear what @davidwrighton has to say on the subject.