Open
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | c1acb35ce2b90f241ccdf3ead79544516c673e5f |
| Compare | 5a99f1defea16d6725a7d8d3be82f52e1671973a |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Linq.Tests.Perf_Enumerable
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Where - Duration of single invocation | 22.45 μs | 25.17 μs | 1.12 | 0.14 | False | |||||
| WhereSelect - Duration of single invocation | 25.21 μs | 27.03 μs | 1.07 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'Payloads
Histogram
System.Linq.Tests.Perf_Enumerable.Where(input: IEnumerable)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.166461521620143 > 23.0847629759359.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/21/2023 4:52:34 PM, 3/27/2023 3:21:55 PM falls between 3/18/2023 7:37:20 PM and 3/27/2023 3:21:55 PM.
IsRegressionStdDev: Marked as regression because -16.32190746559229 (T) = (0 -25274.49533558129) / Math.Sqrt((385782.3349108681 / (29)) + (566100.2267068804 / (32))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (32) - 2, .025) and -0.12827382372454604 = (22401.029611896538 - 25274.49533558129) / 22401.029611896538 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Linq.Tests.Perf_Enumerable.WhereSelect(input: IEnumerable)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.02850032051282 > 26.321914993031495.
IsChangePoint: Marked as a change because one of 1/23/2023 10:43:32 PM, 3/10/2023 8:06:53 PM, 3/21/2023 7:06:30 PM, 3/27/2023 3:21:55 PM falls between 3/18/2023 7:37:20 PM and 3/27/2023 3:21:55 PM.
IsRegressionStdDev: Marked as regression because -13.662921556453544 (T) = (0 -27917.846538973) / Math.Sqrt((580937.908150486 / (30)) + (277564.96424129757 / (31))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (31) - 2, .025) and -0.08974723244575952 = (25618.64412935095 - 27917.846538973) / 25618.64412935095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository