[release/3.1] Enable additional CI testing

[echesakov]

Enable additional CI testing using new Helix queue Windows.10.Arm64v8.Open.

Customer Impact

None

Testing

Validated the change by manually triggering two pipelines coreclr-ci#20210217.2 and coreclr-outerloop#20210217.1

Risk

None - the change affects testing infrastructure only

cc @BruceForstall @JulieLeeMSFT @jeffschwMSFT

[JulieLeeMSFT]

LGTM

[BruceForstall]

It looks good, and it looks like the tests ran, but can you be sure?

The Windows arm64 tests don't show up on the "Tests" results tab:

https://dev.azure.com/dnceng/public/_build/results?buildId=999073&view=ms.vss-test-web.build-test-results-tab
https://dev.azure.com/dnceng/public/_build/results?buildId=999284&view=ms.vss-test-web.build-test-results-tab

Can you find one/some console logs in Kusto (or elsewhere)?

[BruceForstall]

It looks like Windows arm64 Pri-0 tests don't do R2R? Maybe R2R was limited in 3.1 testing?

[echesakov]

It looks good, and it looks like the tests ran, but can you be sure?

The Windows arm64 tests don't show up on the "Tests" results tab:

https://dev.azure.com/dnceng/public/_build/results?buildId=999073&view=ms.vss-test-web.build-test-results-tab
https://dev.azure.com/dnceng/public/_build/results?buildId=999284&view=ms.vss-test-web.build-test-results-tab

Can you find one/some console logs in Kusto (or elsewhere)?

I had exactly the same question earlier today and had already validated that. I should've mentioned this in the PR.

I copied output from work items corresponding to JIT.Regression test collection that ran on the new Helix queue

Test Pri0 Windows_NT arm64 checked

Console log: 'JIT.Regression' from job 7a2f863f-6aec-4ad3-8106-b2d32fd32d07 workitem 27989d34-c287-4aec-8a1a-bb8b89a62ae7 (windows.10.arm64v8.open) executed on machine DDARM64-094

D:\h\w\B459099D\w\AAC209B4\e>set CORE_ROOT=D:\h\w\B459099D\p 

D:\h\w\B459099D\w\AAC209B4\e>set __TestEnv=D:\h\w\B459099D\w\AAC209B4\u\SetStressModes_normal.cmd 

D:\h\w\B459099D\w\AAC209B4\e>set __TestTimeout=600000 

D:\h\w\B459099D\w\AAC209B4\e>type D:\h\w\B459099D\w\AAC209B4\u\SetStressModes_normal.cmd 

D:\h\w\B459099D\w\AAC209B4\e>D:\h\w\B459099D\p\CoreRun.exe D:\h\w\B459099D\p\xunit.console.dll JIT\Regression\JIT.Regression.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=JIT.Regression 
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET Core 3.1.13-ci.21117.1)
  Discovering: JIT.Regression.XUnitWrapper
  Discovered:  JIT.Regression.XUnitWrapper
  Starting:    JIT.Regression.XUnitWrapper
  Finished:    JIT.Regression.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.Regression.XUnitWrapper  Total: 277, Errors: 0, Failed: 0, Skipped: 0, Time: 42.916s

D:\h\w\B459099D\w\AAC209B4\e>set _commandExitCode=0 

Test Pri1 Windows_NT arm64

Console log: 'JIT.Regression' from job e38e2f02-240c-47d4-bcd0-44b81c47ae14 workitem 80413baa-7a02-4d28-9eb9-6c44f4beb8b8 (windows.10.arm64v8.open) executed on machine DDARM64-084

D:\h\w\ACB80965\w\AA3A09A4\e>set CORE_ROOT=D:\h\w\ACB80965\p 

D:\h\w\ACB80965\w\AA3A09A4\e>set __TestEnv=D:\h\w\ACB80965\w\AA3A09A4\u\SetStressModes_no_tiered_compilation.cmd 

D:\h\w\ACB80965\w\AA3A09A4\e>set __TestTimeout=600000 

D:\h\w\ACB80965\w\AA3A09A4\e>type D:\h\w\ACB80965\w\AA3A09A4\u\SetStressModes_no_tiered_compilation.cmd 
set COMPlus_TieredCompilation=0

D:\h\w\ACB80965\w\AA3A09A4\e>D:\h\w\ACB80965\p\CoreRun.exe D:\h\w\ACB80965\p\xunit.console.dll JIT\Regression\JIT.Regression.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=JIT.Regression 
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET Core 3.1.13-ci.21117.1)
  Discovering: JIT.Regression.XUnitWrapper
  Discovered:  JIT.Regression.XUnitWrapper
  Starting:    JIT.Regression.XUnitWrapper
  Finished:    JIT.Regression.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.Regression.XUnitWrapper  Total: 671, Errors: 0, Failed: 0, Skipped: 0, Time: 141.970s

Test Pri1 R2R Windows_NT arm64 checked

Console log: 'JIT.Regression' from job 2e1cfbfe-142d-44c5-8c16-2207653950ae workitem c9e062c6-96a7-46d6-9bd8-4102d6a68a4a (windows.10.arm64v8.open) executed on machine DDARM64-094

D:\h\w\B3CE093D\w\AAF9094F\e>set CORE_ROOT=D:\h\w\B3CE093D\p 

D:\h\w\B3CE093D\w\AAF9094F\e>set RunCrossGen=1 

D:\h\w\B3CE093D\w\AAF9094F\e>set __TestEnv=D:\h\w\B3CE093D\w\AAF9094F\u\SetStressModes_normal.cmd 

D:\h\w\B3CE093D\w\AAF9094F\e>set __TestTimeout=600000 

D:\h\w\B3CE093D\w\AAF9094F\e>type D:\h\w\B3CE093D\w\AAF9094F\u\SetStressModes_normal.cmd 

D:\h\w\B3CE093D\w\AAF9094F\e>D:\h\w\B3CE093D\p\CoreRun.exe D:\h\w\B3CE093D\p\xunit.console.dll JIT\Regression\JIT.Regression.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=JIT.Regression 
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET Core 3.1.13-ci.21117.1)
  Discovering: JIT.Regression.XUnitWrapper
  Discovered:  JIT.Regression.XUnitWrapper
  Starting:    JIT.Regression.XUnitWrapper
  Finished:    JIT.Regression.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.Regression.XUnitWrapper  Total: 671, Errors: 0, Failed: 0, Skipped: 0, Time: 206.354s

I believe the reason why we can't see the test results in AzDO Tests tab is due to

D:\h\w\B459099D\w\AAC209B4\e>if NOT EXIST D:\Users\runner\.vsts-env\Scripts\python.exe (
rmdir /Q /S D:\Users\runner\.vsts-env-tmp  
 rmdir /Q /S D:\Users\runner\.vsts-env  
 C:\python3.7.0\python.exe -m virtualenv --no-site-packages D:\Users\runner\.vsts-env-tmp  
 rename D:\Users\runner\.vsts-env-tmp .vsts-env 
) 
The system cannot find the file specified.
Using base prefix 'C:\\python3.7.0'
New python executable in D:\Users\runner\.vsts-env-tmp\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
Access is denied.

D:\h\w\B459099D\w\AAC209B4\e>D:\Users\runner\.vsts-env\Scripts\python.exe -c "import azure.devops"   || D:\Users\runner\.vsts-env\Scripts\python.exe -m pip install azure-devops==5.0.0b9 
The system cannot find the path specified.
The system cannot find the path specified.

D:\h\w\B459099D\w\AAC209B4\e>D:\Users\runner\.vsts-env\Scripts\python.exe -c "import future"   || D:\Users\runner\.vsts-env\Scripts\python.exe -m pip install future==0.17.1 
The system cannot find the path specified.
The system cannot find the path specified.

I opened https://github.com/dotnet/core-eng/issues/12223

[echesakov]

It looks like Windows arm64 Pri-0 tests don't do R2R? Maybe R2R was limited in 3.1 testing?

Not sure (remember) why it was configured that way.
But Windows arm64 R2R Pri-1 tests are running, shouldn't this be sufficient?

[BruceForstall]

Thanks for double checking they are running; looks good.

But Windows arm64 R2R Pri-1 tests are running, shouldn't this be sufficient?

yeah, that's fine.

[echesakov]

Found out that gcstress, r2r and corefx* pipelines don't use the new queue - updated a bunch of yml files to fix this.
I am going to test them now.

[jeffschwMSFT]

Given this is an infrastructure only change, we can take via tell mode. Please coordinate with Matt Mitchel and Anirudh Agnihotry once we are ready to merge and they can help us with timing. Post merging we can send a tell mode notification. Thanks @echesakovMSFT for prepping this change.

[echesakov]

Validated that the remaining pipelines in coreclr work with the new queue and the results of their runs are expected.

@mmitche @Anipik Please advise when this can be merged

[Anipik]

@mmitche @Anipik Please advise when this can be merged

it will probably next month

[mmitche]

@mmitche @Anipik Please advise when this can be merged

it will probably next month

Yeah when branches open for the next release/.

[JulieLeeMSFT]

Yeah when branches open for the next release/.

@mmitche @Anipik What is the timeline/date in March to open the branch? How can we keep track of it, or would you notify us?

[mmitche]

It's either going to be March 9th or early next week.

[jeffschwMSFT]

@leecow I am going to mark this as servicing-approved just so that we do not get missed on the next opportunity to merge. If there is a concern, please feel free to remove from this infra only change.

[aik-jahoda]

There are failing tests, re-run...

[echesakov]

There are failing tests, re-run...

@aik-jahoda You would observe the same test failures when running against release/3.1 branch. Note that the change doesn't affect platforms other than win-arm64.

[Anipik]

Merging this one in respone to

@aik-jahoda You would observe the same test failures when running against release/3.1 branch. Note that the change doesn't affect platforms other than win-arm64.