[CI] Outerloop Tests workflow is broken

[radical]

This broke with #8802 . Now the runsheets are produced only for couple of test projects. - https://github.com/dotnet/aspire/actions/runs/14501563823/job/40682240457

cc @danmoseley @captainsafia @Youssef1313

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[Youssef1313]

@radical @RussKie Let's see if #8833 gets us on track again.

[RussKie]

The latest run contains all the expected projects:

I can't explain why the previous runs behaves as they did. Closing for now; we can reopen, if necessary.

[Youssef1313]

@RussKie Latest run is from my branch (#8833). If it unblocks VS Code issues, we should be able to get it merged.

[RussKie]

Ah, bugger. I should learn to read :)

[RussKie]

@Youssef1313 I need some extra pair of eyes here. This feels like dotnet test is playing tricks.

Locally I'm on the latest main (7cec38b). When I run the outerloop commands I get the expected number of quarantined tests. However, the same commit, same commands on GHA produce a different result.

"D:\a\aspire\aspire\.dotnet\dotnet.exe" test D:\a\aspire\aspire\artifacts\bin\Aspire.Hosting.Azure.Tests\Release\net8.0\Aspire.Hosting.Azure.Tests.dll --list-tests --logger:"console;verbosity=normal" "--Framework:.NETCoreApp,Version=v8.0" "--settings:D:\a\aspire\aspire\eng\testing\.runsettings"  --filter "quarantined=true"  --blame --blame-hang-timeout 10m --blame-crash >> "D:\a\aspire\aspire\artifacts\log\Release\TestLogs\Aspire.Hosting.Azure.Tests_net8.0_x64.log" 2>&1 >> D:\a\aspire\aspire\artifacts\log\Release\TestLogs\Aspire.Hosting.Azure.Tests_net8.0_x64.log

I have even published all binaries produced by the workflow and ran against those, and I get the list of quarantined tests locally:

PS D:\Development\dotnet-aspire> .\.dotnet\dotnet test "C:\Users\user\Downloads\logs-runsheet (11)\bin\Aspire.Hosting.Azure.Tests\Release\net8.0\Aspire.Hosting.Azure.Tests.dll" --list-tests --filter "quarantined=true" "--Framework:.NETCoreApp,Version=v8.0" "--settings:D:\Development\dotnet-aspire\eng\testing\.runsettings" --blame --blame-hang-timeout 10m --blame-crash

VSTest version 17.12.0 (x64)

The following Tests are available:
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.2+dd36e86129 (64-bit .NET 8.0.14)
[xUnit.net 00:00:00.23]   Discovering: Aspire.Hosting.Azure.Tests
[xUnit.net 00:00:00.66]   Discovered:  Aspire.Hosting.Azure.Tests
    Aspire.Hosting.Azure.Tests.AzureCosmosDBEmulatorFunctionalTests.AddAzureCosmosDB_RunAsEmulator_CreatesDatabase

What am I missing?

[Youssef1313]

The binlog you showed is from CI right? That has category!=failing, but the test has RequiresDocker which isn't available on GHA on Windows, so it makes sense to me that it produces no tests. Even if you run with category!=failing part locally, the result will be different:

aspire/tests/Aspire.TestUtilities/RequiresDockerAttribute.cs

Lines 22 to 24 in 09bcfce

	public static bool IsSupported =>
	!OperatingSystem.IsWindows() ||
	!PlatformDetection.IsRunningOnCI;

[Youssef1313]

This is probably now fixed as #8833 is merged.