[Perf] Linux/x64: 9 Regressions on 6/28/2023 3:57:02 AM #19515
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 2253c7e283988f91d334f1c4e6f3afc2aed69960 |
| Compare | 293b250c9e6195a094422f4dacbd395087bcd5f3 |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.IO.Tests.Perf_File
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CopyToOverwrite - Duration of single invocation | 27.56 μs | 32.62 μs | 1.18 | 0.18 | False | |||||
| CopyTo - Duration of single invocation | 39.35 μs | 45.46 μs | 1.16 | 0.12 | False | |||||
| ReadAllBytes - Duration of single invocation | 16.20 μs | 18.22 μs | 1.12 | 0.15 | False | |||||
| CopyToOverwrite - Duration of single invocation | 28.89 μs | 31.28 μs | 1.08 | 0.17 | False | |||||
| ReadAllBytes - Duration of single invocation | 15.43 μs | 17.49 μs | 1.13 | 0.19 | False | |||||
| CopyTo - Duration of single invocation | 40.51 μs | 45.55 μs | 1.12 | 0.12 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'Payloads
System.IO.Tests.Perf_File.CopyToOverwrite(size: 4096)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.61990997646443 > 29.28305138677812.
IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -15.283514221027811 (T) = (0 -31803.50762640097) / Math.Sqrt((246907.51894742117 / (17)) + (849612.9229249641 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.1296077215142923 = (28154.470813785578 - 31803.50762640097) / 28154.470813785578 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_File.CopyTo(size: 512)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.456976562499996 > 42.695931982421875.
IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -24.686783313626645 (T) = (0 -46004.35817323615) / Math.Sqrt((245108.82263384215 / (17)) + (725139.4048051961 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.13739621551691134 = (40447.082156263896 - 46004.35817323615) / 40447.082156263896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_File.ReadAllBytes(size: 4096)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.21640495155039 > 17.09185064484016.
IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -14.737182241011007 (T) = (0 -18366.554457186652) / Math.Sqrt((181105.3244932657 / (17)) + (142518.06438644725 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.1198080310461967 = (16401.52057136743 - 18366.554457186652) / 16401.52057136743 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_File.CopyToOverwrite(size: 512)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.2785379316325 > 29.35856496717702.
IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -11.172123604781955 (T) = (0 -31271.969931170068) / Math.Sqrt((756068.4685568523 / (17)) + (613855.2322197123 / (18))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (18) - 2, .025) and -0.11129038982001657 = (28140.232487959194 - 31271.969931170068) / 28140.232487959194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_File.ReadAllBytes(size: 512)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.48931581737988 > 16.578717057061507.
IsChangePoint: Marked as a change because one of 5/3/2023 6:00:09 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -11.626793097676737 (T) = (0 -17988.33040855564) / Math.Sqrt((304511.555375814 / (17)) + (322385.985084188 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.13537926344437268 = (15843.455123519587 - 17988.33040855564) / 15843.455123519587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_File.CopyTo(size: 4096)
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.55031560981537 > 42.525360442910205.
IsChangePoint: Marked as a change because one of 5/3/2023 1:06:54 PM, 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -20.573913894248662 (T) = (0 -45888.19165348747) / Math.Sqrt((622613.4003034956 / (17)) + (448797.16478189325 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.12228753414929924 = (40888.08817454343 - 45888.19165348747) / 40888.08817454343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 2253c7e283988f91d334f1c4e6f3afc2aed69960 |
| Compare | 293b250c9e6195a094422f4dacbd395087bcd5f3 |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.IO.Tests.Perf_Directory
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| MoveFiles - Duration of single invocation | 17.23 μs | 19.79 μs | 1.15 | 0.13 | False | |||||
| EnumerateFiles - Duration of single invocation | 52.08 ms | 54.99 ms | 1.06 | 0.22 | False | |||||
| MoveFolders - Duration of single invocation | 18.30 μs | 21.94 μs | 1.20 | 0.24 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_Directory*'Payloads
System.IO.Tests.Perf_Directory.MoveFiles
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.78575193282636 > 18.19557393494834.
IsChangePoint: Marked as a change because one of 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -12.162376738621802 (T) = (0 -19674.031778879333) / Math.Sqrt((391241.87818382034 / (17)) + (228592.75067801014 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.12960023894965328 = (17416.80915114981 - 19674.031778879333) / 17416.80915114981 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_Directory.EnumerateFiles
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.98758923214286 > 54.103728912019236.
IsChangePoint: Marked as a change because one of 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -12.856998789997157 (T) = (0 -56915726.594661176) / Math.Sqrt((2049781171239.9595 / (17)) + (2186566305200.1226 / (20))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (20) - 2, .025) and -0.12146965726388662 = (50751017.85056023 - 56915726.594661176) / 50751017.85056023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_Directory.MoveFolders
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.943064907227086 > 19.65212445400428.
IsChangePoint: Marked as a change because one of 6/27/2023 10:31:09 PM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -17.47738052325529 (T) = (0 -21436.860649437534) / Math.Sqrt((226426.20481472317 / (17)) + (140664.01620814012 / (17))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (17) - 2, .025) and -0.13611265363762234 = (18868.60478210561 - 21436.860649437534) / 18868.60478210561 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository