[Perf] Changes at 11/18/2021 2:30:24 PM #2557
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | d9afc1ecd6f22d836e1d7665847917aa09f580e9 |
| Compare | 6ff57f1935d1cb3396674bec26268a301e5c0628 |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CtorInvoke - Duration of single invocation | 10.94 μs | 8.62 μs | 0.79 | 0.28 | False | |||||
| MatchesSet - Duration of single invocation | 250.38 μs | 195.48 μs | 0.78 | 0.00 | True | |||||
| Date_IsMatch - Duration of single invocation | 280.11 ns | 235.60 ns | 0.84 | 0.01 | True | |||||
| SplitWords - Duration of single invocation | 11.71 μs | 4.14 μs | 0.35 | 0.02 | True | |||||
| ReplaceWords - Duration of single invocation | 11.71 μs | 4.40 μs | 0.38 | 0.02 | True | |||||
| Email_IsNotMatch - Duration of single invocation | 996.33 ns | 933.24 ns | 0.94 | 0.00 | False | |||||
| MatchWord - Duration of single invocation | 297.67 ns | 209.80 ns | 0.70 | 0.26 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: None)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.620712749405087 < 9.019480297699921.
IsChangePoint: Marked as a change because one of 10/22/2021 6:47:49 PM, 11/18/2021 9:28:27 AM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 21.65001200745392 (T) = (0 -8682.692143688411) / Math.Sqrt((193998.8770616192 / (299)) + (53969.205014741514 / (29))) is greater than 1.9672675222597202 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (29) - 2, .975) and 0.1110465678812588 = (9767.319445512449 - 8682.692143688411) / 9767.319445512449 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 195.48354964816258 < 237.547342919597.
IsChangePoint: Marked as a change because one of 10/25/2021 8:49:36 PM, 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 129.5442343079608 (T) = (0 -195732.17845689665) / Math.Sqrt((76473579.10447659 / (299)) + (98462.52797769953 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.2518359371416852 = (261616.65358412685 - 195732.17845689665) / 261616.65358412685 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 235.60126250991235 < 263.9160499148434.
IsChangePoint: Marked as a change because one of 11/4/2021 10:34:48 AM, 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 111.61529698399299 (T) = (0 -235.71335978576178) / Math.Sqrt((50.2699987870205 / (299)) + (1.4603715460492703 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.17867955586653844 = (286.99317235972666 - 235.71335978576178) / 286.99317235972666 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.135653764299016 < 10.941505245752738.
IsChangePoint: Marked as a change because one of 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 419.4567787716533 (T) = (0 -4214.972652037255) / Math.Sqrt((29446.298076475494 / (299)) + (5544.379869962573 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.6167744811244866 = (10998.674264712634 - 4214.972652037255) / 10998.674264712634 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.397003866590758 < 11.11215526544092.
IsChangePoint: Marked as a change because one of 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 401.3292264029626 (T) = (0 -4420.131979485481) / Math.Sqrt((42148.50702588483 / (197)) + (2233.324925409629 / (34))) is greater than 1.9703772833260038 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (197) + (34) - 2, .975) and 0.6029099906503327 = (11131.30996854979 - 4420.131979485481) / 11131.30996854979 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 933.2433918102472 < 942.4189885847665.
IsChangePoint: Marked as a change because one of 11/4/2021 5:49:01 AM, 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 43.05878088452612 (T) = (0 -934.1032545018807) / Math.Sqrt((368.03942541439426 / (299)) + (124.19168859279715 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.09887825773811293 = (1036.6005065610861 - 934.1032545018807) / 1036.6005065610861 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 209.80139246810953 < 286.1482524766606.
IsChangePoint: Marked as a change because one of 11/17/2021 1:22:14 PM, 11/24/2021 10:47:45 PM falls between 11/16/2021 7:52:31 AM and 11/24/2021 10:47:45 PM.
IsImprovementStdDev: Marked as improvement because 63.72220179592395 (T) = (0 -213.71930805394885) / Math.Sqrt((115.49926371870367 / (299)) + (66.43762869476069 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.3132467490347592 = (311.2024701063497 - 213.71930805394885) / 311.2024701063497 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository