Support testing previous non-current .NETCoreApp targeted libraries

[ViktorHofer]

From #54636

from @eerhardt

@ericstj @ViktorHofer - this is another case where we are splitting ".NET Core 3.1" build with ".NET Current" build. (I had to do that in order to use ComWrappers, because that API came in 5.0.) This means we are going to be missing test coverage for all of these COM interop scenarios for System.Drawing.Common v6.0.0 when running on .NET Core 3.1.

from @ericstj

@ViktorHofer what if this was opt-in per test? A test project could decide to add a configuration for netcore3.1-Windows and we could have one of the build legs (either windows or all-configurations) create the helix work items. Would this require more capacity than a single test additional test assembly?

The overall issue isn't specific to Windows, we need coverage on as many OSs that a library supports, as possible. The cost that I'm concerned about isn't observable in the amount of build agents but in the number of Helix clients that accept work items.

I imagine support for testing on older .NETCoreApp configurations could be implemented in the following way in order:

• Downgrade the TestUtilities.csproj to use $(NetCoreAppMinimum).
• Add infra support for running .NETCoreApp tests without a live built testhost (same as for .NETFramework testing) here.
• Add CI legs previous .NET versions that build the respective TFMs and submit jobs to helix for all OSs. Consider adding a new pipeline for that so that it can be scheduled (i.e. nightly).
• Add NetCoreAppMinimum / NetCoreAppPrevious TFMs to the OOB test projects either everywhere or on a per project basis where it makes sense.
• Make sure that the new pipeline is monitored to catch regressions in the older .NETCoreApp TFM assets.

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

From #54636

from @eerhardt

@ericstj @ViktorHofer - this is another case where we are splitting ".NET Core 3.1" build with ".NET Current" build. (I had to do that in order to use ComWrappers, because that API came in 5.0.) This means we are going to be missing test coverage for all of these COM interop scenarios for System.Drawing.Common v6.0.0 when running on .NET Core 3.1.

from @ericstj

@ViktorHofer what if this was opt-in per test? A test project could decide to add a configuration for netcore3.1-Windows and we could have one of the build legs (either windows or all-configurations) create the helix work items. Would this require more capacity than a single test additional test assembly?

The overall issue isn't specific to Windows, we need coverage on as many OSs that a library supports, as possible. The cost that I'm concerned about isn't observable in the amount of build agents but in the number of Helix clients that accept work items.

I imagine support for testing on older .NETCoreApp configurations could be implemented in the following way in order:

• Downgrade the TestUtilities.csproj to netcoreapp3.1 (use $(NetCoreAppMinimum) which will be introduced soon).
• Add infra support for running .NETCoreApp tests without a testhost (same as for .NETFramework testing) here.
• Add two legs (or as an optimization only one leg) for netcoreapp3.1 and net5.0 that build the respective configuration and submit jobs to helix for all OSs. Consider adding a new pipeline for that so that it can be scheduled (i.e. nightly).
• Add netcoreapp3.1/net5.0 TFMs in the OOB test projects either everywhere or on a per project basis where it makes sense (I think the former would make more sense).
• Make sure that the new pipeline is monitored to catch regressions in the older .NETCoreApp TFM assets.

Author:	ViktorHofer
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[ViktorHofer]

from @safern:

@ViktorHofer do we have a lot of projects that would run on previous versions of .NET Core? What if we start small and run them on Ubuntu, Windows and macOS on the build agent without using helix? If there aren't may projects that would want to do this, we could get away without using helix and running them on the build agent.

Every OOB project that today targets $(NetCoreAppCurrent) will need to continue to test for net6.0 when we change main to target net7.0. Basically that means every OOB that isn't .NETFramework specific (95% of all OOBs).

What if we start small and run them on Ubuntu, Windows and macOS on the build agent without using helix?

That was my idea as well but ideally we would have a plan for the long run as well to be able to test for all compatible OSs, i.e. Browser as well. I'm fine with starting on build agents but is there a reason to not use Helix for it if it's just a nightly job?

[safern]

I'm fine with starting on build agents but is there a reason to not use Helix for it if it's just a nightly job?

That's fine as well, however that would be yet another pipeline to take a look at.