TIME OUT - Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release

[thaystg]

We couldn't find any obvious change that could be causing it between these commits:
git diff a82d92c..c8da2fd

Log which the timeout started:
https://dev.azure.com/dnceng/public/_build/results?buildId=1633880&view=results

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[thaystg]

@MattGal did we have any relevant infra change between friday and saturday?

[simonrozsival]

The timeout seems to be related to my PR which was merged on Friday (#65128). I'll try reverting it and see if it helps.

[MattGal]

@MattGal did we have any relevant infra change between friday and saturday?

Aside from serious "world-on-fire" hot-fixes we will not make infrastructure changes on a Friday evening or Saturday morning. If something's going to roll out and affect you, it'll almost always be on a Wednesday in the morning time for the Pacific time zone.

If your investigation does make you think there are infra issues, please feel free to also/instead tag @dotnet/dnceng as sometimes I take vacation days and there are other folks who might reply faster on those days. Including things like the Job GUID (sometimes called Correlation id), Work item friendly name, and why you think we caused your timeout can also speed investigation. Sounds like we're waiting for Simon to check it out first, though.

[vargaz]

This seems to be caused by more assemblies getting AOTed.
In particular, every test project now seems to include TestLibrary.dll, so we end up AOTing it like 2000 extra times. The original lane already took about 2 hours, so these extra AOT steps might push it over the timeout.

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We couldn't find any obvious change that could be causing it between these commits:
git diff a82d92c..c8da2fd

Log which the timeout started:
https://dev.azure.com/dnceng/public/_build/results?buildId=1633880&view=results

Author:	thaystg
Assignees:	-
Labels:	blocking-clean-ci, untriaged, area-Infrastructure-mono, in-pr
Milestone:	-

[vargaz]

The extra TestLibrary dll was added by
572405a#diff-13390079908954088047032dd93629fec2bcfd68749c2c7422c9ed60511ea108L73
Which adds TestLibrary to every test project.

[lewing]

cc @trylek