Slow "Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release"

[EgorBo]

Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release takes around 2.5H to finish.

There are some interesting anomalies in the logs, e.g.:

(I checked various runs)

It says that prejitting of a single managed assembly Microsoft.Win32.SystemEvents.dll takes almost 10 minutes 😮 (mostly in LLVM's opt+llc)

I parsed the output into an excel table:

is it possible to move some libs/tests to the outerloop? e.g. JIT/Methodical/MDArray/GaussJordan/classarr_cs_do/classarr_cs_do test. And I guess we need to figure out what exactly makes Microsoft.Win32.SystemEvents.dll so long to prejit - there are not much stuff in it.

cc @akoeplinger @vargaz @steveisok

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release takes around 2.5H to finish.

There are some interesting anomalies in the logs, e.g.:

(I checked various runs)

It says that prejitting of a single managed assembly Microsoft.Win32.SystemEvents.dll takes almost 10 minutes 😮 (mostly in LLVM's opt+llc)

I parsed the output into an excel table:

is it possible to move some libs/tests to the outerloop? e.g. JIT/Methodical/MDArray/GaussJordan/classarr_cs_do/classarr_cs_do test. And I guess we need to figure out what exactly makes Microsoft.Win32.SystemEvents.dll so long to prejit - there are not much stuff in it.

cc @akoeplinger @vargaz @steveisok

Author:	EgorBo
Assignees:	-
Labels:	untriaged, area-Infrastructure-mono
Milestone:	-

[steveisok]

Adding @SamMonoRT

[vargaz]

Yes, these are very slow, they run opt+llc on unlinked assemblies.

[agocke]

This test run is pretty regularly timing out -- can we get someone to investigate if the slowness is a bug, or if we need to adjust the timeout?

[SamMonoRT]

This test run is pretty regularly timing out -- can we get someone to investigate if the slowness is a bug, or if we need to adjust the timeout?

This PR (#66157) should help ease the timeouts seen in last couple weeks. Even with that fix, the lane is 2.5+hrs long. Still discussing this, but we might possibly 1. want to exclude certain long running tests as part of PR runs in this lane, 2. Extend the timeout to stabilize CI in the short term

[agocke]

@SamMonoRT which PR?

[SamMonoRT]

#66157

[agocke]

Looks like that resolved the problem. I'm going to close this out for now.