Add CI support for testing APX

[BruceForstall]

Add testing support in the CI system for testing APX.

There are three basic phases:

• [DONE: Enable SuperPMI replay for APX ISA testing #113843] Before we have any APX hardware in the CI, what testing can we do? Assertion checking of the new EGPR/REX2/EVEX/If-Convert code paths using an altjit with those features enabled and a SPMI replay would be useful for assertion checking.
• When there are a few APX machines in CI, we can add limited, targeted executable testing.
• Eventually, if a large number or all CI test machines are APX enabled, we can run the full set of testing (including stress) with APX, but then we will also need to disable APX to ensure that scenario still works. Perf testing will also need to include APX.

There exists a runtime-coreclr jitstress-isas-avx512 pipeline (https://dev.azure.com/dnceng-public/public/_build?definitionId=235). Given all our CI machines are AVX-512 capable, do we still need this pipeline? Can this pipeline form a template to create a runtime-coreclr jitstress-isas-apx pipeline?

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[MichalPetryka]

1. Before we have any APX hardware in the CI, what testing can we do?

Could there be a test pipeline running with Intel® Software Development Emulator (Intel® SDE) that has all ISAs emulated to be supported and runs the tests there?

[BruceForstall]

Could there be a test pipeline running with Intel® Software Development Emulator (Intel® SDE) that has all ISAs emulated to be supported and runs the tests there?

It might be possible, but it is preferable to avoid this due to cost (development cost, runtime cost) and the almost inevitable oddities of the emulator. We would prefer to just wait until hardware is available.

[kunalspathak]

Just to make sure we get coverage on JitStressRegs mentioned in #112239 (comment)

[JulieLeeMSFT]

Assigning to @anthonycanino. Please feel free to reassign.

[DeepakRajendrakumaran]

@BruceForstall Considering that force enabling APX will result in code that cannot be executed with existing machines, a pipeline similar to jitstress_isas_x86_evex might not work since this seem to be running actual tests. I was thinking of adding a superpmi pipeline using altjit to verify no asserts happen till we have machines we can execute APX code on. Internally we use sde for some testing but I don't think that's a great option for CI.

Assuming I want to add a superpmi replay pipeline for APX named superpmi-replay-apx, what would I need to add on top of this ?. This is just a copy for existing superpmi-replay. The only things I've modified are the display name and job name. It's meant to trigger(for experimental purposes) when there is a change in JIT for a PR. I have made a superficial change in JIT folder to trigger it. I can't quite figure out what else is required to add and trigger this for this PR

[BruceForstall]

It probably needs changes in src\coreclr\scripts\superpmi-replay.proj and superpmi_replay.py.

However, it probably makes sense to delegate this work to Microsoft, since for security reasons only Microsoft can create a new pipeline on which this test code will run.