Logging tweaks for playwright

[HaoK]

No description provided.

[HaoK]

@dougbu hrm looks like something weird is going on where we are running blazor templates twice now (once in Windows Template job) and again in helix, I thought BuildHelixPayload=true was supposed to automatically SkipTests, did some refactoring change the expectations there? I only noticed it while investigating logs, but figure we should avoid running these tests twice

[dougbu]

we are running blazor templates twice now (once in Windows Template job) and again in helix

This is happening because the "Test Templates" step in the "Test: Templates - Windows Server 2016 x64" job doesn't specify /p:SkipHelixReadyTests=true. That wasn't relevant before but is now.

[HaoK]

Actually do we actually need the test templates job at all now that blazor templates are on helix, this was the last bucket of tests in project templates I believe, so maybe I can just get rid of that job entirely?

[HaoK]

We had split the blazor templates out from project template originally so they could continue running on azdo while moving the rest of them onto helix, so I don't think we need it any longer

[dougbu]

@HaoK we have a lot of individual project template tests that are skipped on at least some Helix queues. Need the templates test job to cover them. One reason is the cert / credential issues we've had in Helix testing.

[HaoK]

@dougbu so its intentional that we want to run them in both helix and in test templates job?

[dougbu]

so its intentional that we want to run them in both helix and in test templates job?

No, as I said, the build step needs /p:SkipHelixReadyTests=true. My pushback was just on removing the job entirely.

[HaoK]

I don't see how we'd run any tests in this job if we turn on SKipHelixReadyTests, this would stop running both ProjectTemplates and BlazorTemplates in the test templates job so we'd have a no-op azdo job no? What tests would still run in this job if I set this flag?

From Directory.Build.targets:

    <SkipTests Condition="'$(SkipHelixReadyTests)' == 'true' AND '$(BuildHelixPayload)' == 'true'">true</SkipTests>

[HaoK]

To clarify, today we run Project templates in both helix and azdo, if I set this flag we'd only run the tests in helix... So your statement that we want azdo coverage relies on this 'bug'/feature where we run on both jobs

[dougbu]

I don't see how we'd run any tests in this job if we turn on SKipHelixReadyTests

You're right. $(BuildHelixPayload) and $(SkipHelixReadyTests) are too coarse-grained.

We probably already have a fair number of tests in other Helix-ready projects that never execute because they use [SkipOnHelix(...)]. That's a bug and I think I mentioned it at some point. One solution would be to add a new property that allows us to run only [SkipOnHelix(...)] tests on the build agents.

Another (ugly) option would be to continue not running those tests at all. Might as well remove them or treat them as manual-only tests☹️

[HaoK]

Right well SkipOnHelix effectively is just a SkipTest except for Project Templates and a few handful of other tests projects, I'm not trying to resolve that as part of this change, but I was just asking if we want to continue running templates tests on both platforms. Which I think just translates down to whether we want to keep this job around or not now

[dougbu]

For now, let's keep the duplication. But please file an issue about removing the dupes and about the tests e.g. Microsoft.AspNetCore.Mvc.FunctionalTests.HtmlGenerationTest.HtmlGenerationWebSite_GeneratesExpectedResultsNotReadyForHelix(...) that (I'm pretty sure) never run.

[HaoK]

Ok I'll file an issue for the duplication in general, there's already an issue for those specific tests since I did that work for those projects: #29597

[HaoK]

Rolling logging changes into #30907