[iOS][tvOS] Several Library tests crashed on CI but don't crash locally

[mdh1418]

When running library tests for mobile, many tests are crashing on CI, but they are not crashing locally. This makes it difficult to reproduce as they seem to be flakey.

The suites in question are skipped in src/libraries/tests.proj under a comment that references this issue and contains Crashes on CI but passes locally

runtime/src/libraries/tests.proj

Lines 191 to 222 in f4e01f5

	<!-- Crashes on CI but passes locally https://github.com/dotnet/runtime/issues/52615 -->
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)Microsoft.Extensions.Http/tests/Microsoft.Extensions.Http.Tests/Microsoft.Extensions.Http.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)Microsoft.Extensions.Logging/tests/Common/Microsoft.Extensions.Logging.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.Compression/tests/System.IO.Compression.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.Compression.Brotli/tests/System.IO.Compression.Brotli.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.Packaging/tests/System.IO.Packaging.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Private.Uri/tests/UnitTests/System.Private.Uri.Unit.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Private.Xml/tests/XmlResolver/System.Xml.XmlResolver.Tests/System.Xml.XmlResolver.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Security.Cryptography.Csp/tests/System.Security.Cryptography.Csp.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Threading.Tasks.Dataflow/tests/System.Threading.Tasks.Dataflow.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Threading.Tasks/tests/System.Threading.Tasks.Tests.csproj" />
	</ItemGroup>
	<ItemGroup Condition="('$(TargetOS)' == 'tvOS' or '$(TargetOS)' == 'tvOSSimulator') and '$(RunDisablediOSTests)' != 'true'">
	<!-- https://github.com/dotnet/runtime/issues/50589 -->
	<ProjectExclusions Include="$(RepoRoot)\src\tests\FunctionalTests\tvOS\Simulator\AOT\tvOS.Simulator.Aot.Test.csproj" />
	<!-- Crashes on CI but passes locally https://github.com/dotnet/runtime/issues/52615 -->
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.Compression.Brotli/tests/System.IO.Compression.Brotli.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.IO.Compression.ZipFile/tests/System.IO.Compression.ZipFile.Tests.csproj" />
	</ItemGroup>
	<ItemGroup Condition="('$(TargetOS)' == 'MacCatalyst') and '$(RunDisablediOSTests)' != 'true'">
	<!-- Crashes randomly during test runs https://github.com/dotnet/runtime/issues/52460 -->
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Threading.Tasks\tests\System.Threading.Tasks.Tests.csproj" />
	<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Threading.Tasks.Dataflow\tests\System.Threading.Tasks.Dataflow.Tests.csproj" />
	<!-- Crashes on CI but passes locally https://github.com/dotnet/runtime/issues/52615 -->
	<ProjectExclusions Include="$(RepoRoot)\src\tests\FunctionalTests\iOS\Simulator\Interpreter\iOS.Simulator.Interpreter.Test.csproj" />
	<ProjectExclusions Include="$(RepoRoot)\src\tests\FunctionalTests\iOS\Simulator\InvariantCultureOnlyMode\iOS.Simulator.InvariantCultureOnlyMode.Test.csproj" />

Based on buildIDs
00f5708a-f5b1-4f23-b2db-a681d955549f
fa0bb039-1f51-4c30-b9c7-83a0146bed57
52cdf27e-95e3-4440-a93c-0f8df6b696eb
8ee9cad7-4458-4d78-bc1a-18dcad390581
be5e7364-04d4-43b0-8baf-d03fa8e84819

iOS and tvOS
System.IO.Compression.Tests.log
System.Security.Cryptography.Csp.Tests.log
System.Threading.Tasks.Dataflow.Tests.log
System.Threading.Tasks.Tests.log

tvOS
System.Diagnostics.Process.Tests.log

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[marek-safar]

There is no detailed info about what crashed

[mdh1418]

@marek-safar Sorry, I just updated the issue with a few more logs. There were many test suites that were crashing on CI but passing locally and at the time of issue creation, I was still in the process of grabbing all the flakes. There are 15 problematic test suites on iOS/tvOS/MacCatalyst and there are 19 problematic test suites on Android last I checked.

[steveisok]

@mdh1418 I'm going to close this because the logs don't tell us much. If you can, run them against CI again and see if you get the same results. At the very least, the logging should be more detailed since there has been xharness improvements.