Tracking issue for "WORKLOAD TIMED OUT"

[steveharter]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build	Definition	Test	Pull Request
749807	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#105233
748937	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
748618	dotnet/runtime	System.Numerics.Tensors.Net8.Tests.WorkItemExecution
748590	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution
748320	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#105167
747478	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
748081	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
747944	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#105153
747724	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#104962
747439	dotnet/runtime	System.Numerics.Tensors.Net8.Tests.WorkItemExecution
746875	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#104856
746673	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#104848
746336	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
746053	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#103755
745979	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
745513	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
743450	dotnet/runtime	JIT_opt.WorkItemExecution
743225	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
743222	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
740287	dotnet/runtime	System.Numerics.Tensors.Net8.Tests.WorkItemExecution
739967	dotnet/runtime	JIT_opt.WorkItemExecution
739577	dotnet/runtime	Loader_classloader.WorkItemExecution
739539	dotnet/runtime	Loader_classloader.WorkItemExecution
737515	dotnet/runtime	JIT_opt.WorkItemExecution
735679	dotnet/runtime	System.Numerics.Tensors.Net8.Tests.WorkItemExecution
735671	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#103585
735197	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
734188	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution
727973	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
728292	dotnet/runtime	JIT_opt.WorkItemExecution
727276	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution
721722	dotnet/runtime	System.ValueTuple.Tests.WorkItemExecution
720435	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#102403

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	19	33

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command..",
  "BuildRetry": false
}

Author:	steveharter
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, Known Build Error
Milestone:	-

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build	Definition	Test	Pull Request
406029	dotnet/runtime	Loader_classloader.WorkItemExecution	#91261
406036	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
405860	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
405505	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
405371	dotnet/runtime	JIT_opt.WorkItemExecution
405156	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
405201	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91753
405175	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91913
404815	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91498
404502	dotnet/runtime	JIT_opt.WorkItemExecution
403846	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
404006	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91964
403649	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
403432	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91945
403331	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#91910
403278	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
402917	dotnet/runtime	JIT_opt.WorkItemExecution	#91925
386099	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#90764
402796	dotnet/runtime	JIT_opt.WorkItemExecution
402586	dotnet/runtime	JIT_opt.WorkItemExecution
402622	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91915
402416	dotnet/runtime	JIT_opt.WorkItemExecution
402225	dotnet/runtime	JIT_opt.WorkItemExecution
402136	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
402090	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#90436
401838	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
401993	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91892
398263	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91690
401505	dotnet/runtime	JIT_opt.WorkItemExecution	#91042
401473	dotnet/runtime	JIT_opt.WorkItemExecution
401430	dotnet/runtime	JIT_opt.WorkItemExecution	#91788
401363	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91671
401270	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
401274	dotnet/runtime	JIT_opt.WorkItemExecution	#91808
401015	dotnet/runtime	JIT_opt.WorkItemExecution
400995	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91482
400925	dotnet/runtime	JIT_opt.WorkItemExecution	#91831
400893	dotnet/runtime	JIT_opt.WorkItemExecution
400630	dotnet/runtime	JIT_opt.WorkItemExecution
400465	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91828
400257	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
400363	dotnet/runtime	JIT_opt.WorkItemExecution	#91468
400157	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91753
399738	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91790
399748	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91792
399536	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91785
399499	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
399419	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91774
399275	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
399203	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91713
398919	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
398113	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91659
398621	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91520
398510	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
398372	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91019
398117	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
398093	dotnet/runtime	JIT_opt.WorkItemExecution	#91714
398051	dotnet/runtime	JIT_opt.WorkItemExecution	#91520
398047	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91520
397851	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
397767	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91703
397525	dotnet/runtime	JIT_opt.WorkItemExecution
397329	dotnet/runtime	JIT_opt.WorkItemExecution
397233	dotnet/runtime	JIT_opt.WorkItemExecution	#91684
397052	dotnet/runtime	JIT_opt.WorkItemExecution	#91458
397028	dotnet/runtime	JIT_opt.WorkItemExecution	#91290
396987	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91290
396849	dotnet/runtime	JIT_opt.WorkItemExecution	#91019
396755	dotnet/runtime	JIT_opt.WorkItemExecution	#91660
396634	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
396691	dotnet/runtime	JIT_opt.WorkItemExecution
396584	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91542
396520	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
396111	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
396177	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91290
395791	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91458
395609	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91542
395438	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91458
395416	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
395450	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91536
395268	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
395186	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
394754	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91116
394959	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91536
394722	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91542
394654	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
394471	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
394475	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91524
394436	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91520
394347	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
394281	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
394161	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91502
394046	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#91497
393987	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
393980	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
393667	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
393578	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
393552	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#91469
393434	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
393189	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#91458
Displaying 100 of 246 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	55	246

Author:	steveharter
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, untriaged, os-ios, os-tvos, Known Build Error
Milestone:	-

[steveisok]

This is a variation of #82637

[steveisok]

@akoeplinger something is wrong w/ our classification when there is a crash in a test suite.

Check out https://dev.azure.com/dnceng-public/public/_build/results?buildId=494600&view=ms.vss-test-web.build-test-results-tab&runId=11481726&resultId=179767 for example. The net.dot log is huge (350MB) and seems to be logging a crash over and over again.

[akoeplinger]

The classification itself is correct, we're timing out because of the crash log loop. I see a potential reason for this, discussing it on Discord.