[Perf] Linux/x64: 3 Regressions on 11/8/2022 2:29:58 AM #9760
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | bfa4812aec806f09f1e2927939ba98118e5e5bf7 |
| Compare | f33badf332c43c0d325e8c802466ea5fd362fb91 |
| Diff | Diff |
Regressions in System.IO.Tests.StreamReaderReadLineTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ReadLine - Duration of single invocation | 3.05 ms | 4.78 ms | 1.57 | 0.07 | False | |||||
| ReadLine - Duration of single invocation | 1.49 ms | 1.98 ms | 1.33 | 0.04 | False | |||||
| ReadLine - Duration of single invocation | 3.16 ms | 6.27 ms | 1.98 | 0.11 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*'Related Issues
Regressions
Improvements
Payloads
Histogram
Edge Detector Info
System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1])
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.781691966517857 > 3.2166848428125.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM.
IsRegressionStdDev: Marked as regression because -57.09251996792648 (T) = (0 -4600162.224617006) / Math.Sqrt((4605864924.373739 / (10)) + (8595748692.52886 / (36))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (36) - 2, .025) and -0.48856679159305294 = (3090329.739046474 - 4600162.224617006) / 3090329.739046474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 8])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.9781770828125 > 1.5480122912478145.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM.
IsRegressionStdDev: Marked as regression because -89.13452586937301 (T) = (0 -1981355.6460121113) / Math.Sqrt((221593302.941373 / (10)) + (424404172.5873213 / (36))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (36) - 2, .025) and -0.355225713472459 = (1462011.5500430819 - 1981355.6460121113) / 1462011.5500430819 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0])
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 6.270424681089743 > 3.330590737103366.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 11/14/2022 6:23:13 PM falls between 11/5/2022 10:22:58 PM and 11/14/2022 6:23:13 PM.
IsRegressionStdDev: Marked as regression because -143.24380047732507 (T) = (0 -6254404.661465553) / Math.Sqrt((964507010.0760417 / (10)) + (13373204433.892677 / (37))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (37) - 2, .025) and -0.961104183815278 = (3189226.106946429 - 6254404.661465553) / 3189226.106946429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository