Understand what is leading to schema mismatches for PGO

[AndyAyersMS]

#49793 disabled an assert in EfficientEdgeCountReconstructor::Propagate() that was checking for schema mismatches. Re-enabling this leads to about 4600 mismatch reports from recent SPMI collections.

We should understand what is behind these mismatches, as they cause us to throw away PGO data and miss out on optimization opportunities. Possible causes:

• IL skew between a static PGO collection and consumption
• JIT initial flow graph consumption varying between Tier0 and Tier1
• Inconsistent hashing (the runtime uses an IL hash as the PGO data lookup key)
• ... ?

Initial focus should be on the dynamic case where we can be fairly certain that the IL is the same and that mismatches should generally never happen.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AndyAyersMS]

For the dynamic PGO cases (with SPMI) we should have both the Tier0 and the Tier1 compilations of a method in the collection. However there is no explicit link between the two. So probably simplest to find a method with a distinctive name and hope there are just a few instances.

It would also be good to track down a mismatch in a live run, if possible.

For static PGO mismatches we might want to tighten up the compatibility check, though the outcome is similar: we'd end up dropping the data. We still might prefer this as it seems possible presenting unrelated class profiles to the devirtualization machinery might have unexpected effects.

Another option to explore is whether we could make better use of "stale" PGO data; however, this seems tricky, unless we can serialize or reconstruct the flowgraph (plus possibly other breadcrumbs) from the build where the data was collected.

All the profile data is keyed to IL offsets currently. That makes it fairly fragile. We could try and find a keying scheme that might be able to survive minor edits -- say block numbers and call site indexes / descriptors.

[jakobbotsch]

All the profile data is keyed to IL offsets currently. That makes it fairly fragile. We could try and find a keying scheme that might be able to survive minor edits -- say block numbers and call site indexes / descriptors.

I'm wondering if we could go all the way back to source code line numbers when those are available. At least the AutoFDO paper seems to suggest that this is fairly robust for C++. Not sure if we can access the PDBs like this during JIT, though.

[AndyAyersMS]

For the schema mismatches: I modified the jit so that when it does Tier0 instrumentation, it optionally asks the runtime for the pgo data back right after allocating the schema, and then compares the submitted and returned schemas.

This tests round-tripping the schema through the runtime. While that may seem trivial, the schema is stored in compressed format, so this check actually verifies the compression/decompression and lookup logic in the runtime.

With this I am able to see schema match failures running a checked jit in ASP.NET scenarios -- though only scenarios where some R2R code is also in play. If R2R is disabled, then the schemas always match. I have SPMI collections for the runs where mismatches happened, so should be able to replay what the jit sees. Will post an update shortly.

[AndyAyersMS]

After analyzing further, the schema mismatches are from schemas captured in R2R images, matching up what the jit expects to see at Tier1.

Here's one problematic method: System.Linq.Enumerable+<OfTypeIterator>d__61`1<__Canon>.MoveNext()
(here taken from 6.0.0-preview.5.21227.4)

R2R dump shows the following PGO payload:

bool System.Linq.Enumerable+<OfTypeIterator>d__61`1<__Canon>.MoveNext()
Handle: 0x0600026A
Rid: 618
EntryPointRuntimeFunctionId: 483
Number of RuntimeFunctions: 2

PGO info:
ILOffset: 0 InstrumentationKind: NumRuns Other: 10 Count: 1
ILOffset: 18 InstrumentationKind: EdgeIntCount Other: 105 Count: 1
21798
ILOffset: 22 InstrumentationKind: EdgeIntCount Other: 28 Count: 1
440
ILOffset: 28 InstrumentationKind: EdgeIntCount Other: 0 Count: 1
591
ILOffset: 52 InstrumentationKind: EdgeIntCount Other: 105 Count: 1
12498
ILOffset: 72 InstrumentationKind: EdgeIntCount Other: 142 Count: 1
35301
ILOffset: 118 InstrumentationKind: EdgeIntCount Other: 142 Count: 1
21752
ILOffset: 135 InstrumentationKind: EdgeIntCount Other: 135 Count: 1
0
ILOffset: 142 InstrumentationKind: EdgeIntCount Other: 0 Count: 1
57053

but looking at method IL, offsets 22 (0x16) and 28 (0x1c) are not valid IL offsets.

IL_0010  de 7c             leave.s      124 (IL_008e)
IL_0012  02                ldarg.0     
IL_0013  15                ldc.i4.m1   
IL_0014  7d 2a 02 00 0a    stfld        0xA00022A
IL_0019  02                ldarg.0     
IL_001a  02                ldarg.0     
IL_001b  7b 2e 02 00 0a    ldfld        0xA00022E
IL_0020  6f 1d 02 00 0a    callvirt     0xA00021D
IL_0025  7d 2f 02 00 0a    stfld        0xA00022F
IL_002a  02                ldarg.0     

During instrumentation the jit produces the following schema:

[4] New probe for BB04 -> BB08 [source]   (0x12/0x69)  **  (1)
    class probe 0x20
    class probe 0x20
[2] New probe for BB05 -> BB08 [edge]     (0x34/0x69)  **  (4)
    class probe 0x34
    class probe 0x34
[3] New probe for BB06 -> BB11 [source]   (0x48/0x83)  **  (5)
    class probe 0x6f
    class probe 0x6f
[1] New probe for BB09 -> BB11 [source]   (0x76/0x8E)  **  (6)
[0] New probe for BB11 -> BB01 [source]   (0x83/0x00)  **  (7)
[5] New probe for BB10 -> BB10 [source]   (0x87/0x87)  **  (8)

So there are 6 edge probes and 3 class probes. From the R2R we see 8 edge probes.

Now it could be the version of this method that was instrumented has different IL. I don't know how to check this. But the source for this method has not changed too much recently:

runtime/src/libraries/System.Linq/src/System/Linq/Cast.cs

Lines 21 to 30 in 01b7e73

	private static IEnumerable<TResult> OfTypeIterator<TResult>(IEnumerable source)
	{
	foreach (object? obj in source)
	{
	if (obj is TResult result)
	{
	yield return result;
	}
	}
	}

So something seems to be going wrong with this method either in ETL->MIBC or MIBC->R2R processing. Possibly that first class probe at IL offset 0x20 (which has two schema records) gets munged into edge records, and the other two class probes get dropped?

Or perhaps we're instrumenting a 5.0 version...? I'll see if I can find the relevant data in the profile stream.

cc @davidwrighton

[AndyAyersMS]

The 5.0.5 System.Linq assemblies do not appear to have System.Linq.Enumerable+<OfTypeIterator>d__61 (they have ...d__62).
Modulo the type name change the 5.0.5 IL for MoveNext is identical with 6.0 preview 5's.

So that seems to suggest that the IL that was instrumented should have been the same IL that is getting optimized, and so the schemas should match.

[AndyAyersMS]

Here's the MIBC data

      "Method": "[System.Linq]System.Linq.Enumerable+<OfTypeIterator>d__61`1<System.__Canon>.MoveNext()",
      "InstrumentationData": [
        {
          "ILOffset": 0,
          "InstrumentationKind": "NumRuns",
          "Other": 10
        },
        {
          "ILOffset": 18,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 105,
          "Data": 21798
        },
        {
          "ILOffset": 22,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 28,
          "Data": 440
        },
        {
          "ILOffset": 23,
          "InstrumentationKind": "TypeHandleHistogramCount",
          "Other": -1073741824,
          "Data": 440
        },
        {
          "ILOffset": 23,
          "InstrumentationKind": "TypeHandleHistogramTypeHandle",
          "Other": -1073741824,
          "Data": [
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.IDiagnosticsSender>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[S.P.CoreLib]System.Collections.Generic.Dictionary`2+KeyCollection+Enumerator<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackageProvider,System.Threading.Tasks.Task`1<System.Collections.Generic.IReadOnlyList`1<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.IDiagnosticsSender>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[S.P.CoreLib]System.Collections.Generic.Dictionary`2+KeyCollection+Enumerator<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackageProvider,System.Threading.Tasks.Task`1<System.Collections.Generic.IReadOnlyList`1<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<Microsoft.DotNet.Cli.CommandLine.AppliedOption,Microsoft.DotNet.Cli.CommandLine.Option>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.IDiagnosticsSender>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[S.P.CoreLib]System.Collections.Generic.Dictionary`2+KeyCollection+Enumerator<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackageProvider,System.Threading.Tasks.Task`1<System.Collections.Generic.IReadOnlyList`1<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>",
            "[System.Linq]System.Linq.Enumerable+WhereEnumerableIterator`1<Microsoft.TemplateEngine.Abstractions.Mount.IFileSystemInfo>"
          ]
        },
        {
          "ILOffset": 28,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 0,
          "Data": 591
        },
        {
          "ILOffset": 32,
          "InstrumentationKind": "TypeHandleHistogramCount",
          "Other": -1073741824,
          "Data": 21798
        },
        {
          "ILOffset": 32,
          "InstrumentationKind": "TypeHandleHistogramTypeHandle",
          "Other": -1073741824,
          "Data": [
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.OrderedEnumerable`2<NuGet.Configuration.SettingItem,int32>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<Microsoft.AspNetCore.Mvc.ApplicationModels.IApplicationModelConvention>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<Microsoft.AspNetCore.Mvc.ApplicationModels.IApplicationModelConvention>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<Microsoft.AspNetCore.Mvc.ApplicationModels.IApplicationModelConvention>",
            "[S.P.CoreLib]System.Collections.Generic.List`1<object>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Collections.ReadOnlyCollection`1<Microsoft.Build.Construction.ProjectElement>",
            "[System.Linq]System.Linq.OrderedEnumerable`2<NuGet.Configuration.SettingItem,int32>"
          ]
        },
        {
          "ILOffset": 52,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 105,
          "Data": 12498
        },
        {
          "ILOffset": 58,
          "InstrumentationKind": "TypeHandleHistogramCount",
          "Other": -1073741824,
          "Data": 47799
        },
        {
          "ILOffset": 58,
          "InstrumentationKind": "TypeHandleHistogramTypeHandle",
          "Other": -1073741824,
          "Data": [
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>"
          ]
        },
        {
          "ILOffset": 72,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 142,
          "Data": 35301
        },
        {
          "ILOffset": 111,
          "InstrumentationKind": "TypeHandleHistogramCount",
          "Other": -1073741824,
          "Data": 69551
        },
        {
          "ILOffset": 111,
          "InstrumentationKind": "TypeHandleHistogramTypeHandle",
          "Other": -1073741824,
          "Data": [
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[S.P.CoreLib]System.SZArrayEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<Newtonsoft.Json.Linq.JToken>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[S.P.CoreLib]System.SZArrayEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[Microsoft.Build]Microsoft.Build.Construction.ProjectElementContainer+ProjectElementSiblingEnumerable+ProjectElementSiblingEnumerator",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<Newtonsoft.Json.Linq.JToken>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<object>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[S.P.CoreLib]System.Collections.Generic.List`1+Enumerator<NuGet.Configuration.SettingItem>",
            "[System.Linq]System.Linq.Enumerable+SelectEnumerableIterator`2<System.Xml.Linq.XElement,NuGet.Configuration.SettingBase>"
          ]
        },
        {
          "ILOffset": 118,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 142,
          "Data": 21752
        },
        {
          "ILOffset": 135,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 135,
          "Data": 0
        },
        {
          "ILOffset": 142,
          "InstrumentationKind": "EdgeIntCount",
          "Other": 0,
          "Data": 57053
        }
      ]
    },

[AndyAyersMS]

From the MIBC I wonder if some other method's data is getting merged in here?

Also note in the MIBC there are class probes at offsets 32/58/111 which match the jit schema, but that don't make it to the R2R version. I suppose this happens because the type handles refer to unknown classes?

So only the offset 23 class handle table is suspicious... both because it does not seem to match up with jit instrumentation, and because when pulled into the R2R format it gets "converted" into a spurious edge probe.

I looked at the "constituent" MIBCs (for hello world, etc) and they all show this offset 23 entry.

[AndyAyersMS]

Not sure if this is what causes the problems above, but it looks to be wrong:

runtime/src/coreclr/tools/Common/Pgo/PgoFormat.cs

Lines 505 to 514 in 907fc3e

	public bool Equals(PgoSchemaElem x, PgoSchemaElem y)
	{
	if (x.ILOffset != y.ILOffset)
	return false;
	if (x.InstrumentationKind != y.InstrumentationKind)
	return false;
	if (x.InstrumentationKind != y.InstrumentationKind && SchemaMergesItemsWithDifferentOtherFields(x.InstrumentationKind))
	return false;
	return true;
	}