[Perf] Linux/x64: 23 Improvements on 5/1/2023 6:56:14 PM #17280
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 0c419b05455970ea4b009c14984785b59ba7b624 |
| Compare | da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 1.04 ms | 650.10 μs | 0.62 | 0.05 | False | |||||
| Count - Duration of single invocation | 119.40 μs | 64.30 μs | 0.54 | 0.10 | False | |||||
| Count - Duration of single invocation | 323.22 μs | 91.52 μs | 0.28 | 0.00 | False | |||||
| Count - Duration of single invocation | 1.21 ms | 855.05 μs | 0.71 | 0.05 | False | |||||
| Count - Duration of single invocation | 120.88 μs | 67.67 μs | 0.56 | 0.10 | False | |||||
| Count - Duration of single invocation | 117.56 μs | 65.45 μs | 0.56 | 0.11 | False | |||||
| Count - Duration of single invocation | 417.39 μs | 121.32 μs | 0.29 | 0.02 | False | |||||
| Count - Duration of single invocation | 118.22 μs | 66.40 μs | 0.56 | 0.11 | False | |||||
| Count - Duration of single invocation | 89.65 μs | 56.74 μs | 0.63 | 0.19 | False | |||||
| Count - Duration of single invocation | 412.24 μs | 123.66 μs | 0.30 | 0.02 | False | |||||
| Count - Duration of single invocation | 644.83 μs | 431.54 μs | 0.67 | 0.01 | False | |||||
| Count - Duration of single invocation | 90.21 μs | 57.64 μs | 0.64 | 0.20 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: None)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 650.1043796666668 < 0.9868450692903645.
IsChangePoint: Marked as a change because one of 2/25/2023 2:33:42 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 198.60808324747325 (T) = (0 -651036.6205) / Math.Sqrt((991086091.3318729 / (299)) + (1738145.942667662 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.38423066142304396 = (1057273.5271368767 - 651036.6205) / 1057273.5271368767 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: None)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 64.30106613756615 < 113.3891255701336.
IsChangePoint: Marked as a change because one of 3/28/2023 7:07:52 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 119.94138671842958 (T) = (0 -63948.96927626607) / Math.Sqrt((37860021.858572245 / (299)) + (247944.39947474137 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.48424468537767246 = (123990.90705074754 - 63948.96927626607) / 123990.90705074754 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 91.51889030355544 < 306.9789676902636.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 1175.2105311635912 (T) = (0 -91709.64405528648) / Math.Sqrt((678879.9892190812 / (299)) + (72773.9875989409 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.7158702116286109 = (322773.7738480691 - 91709.64405528648) / 322773.7738480691 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 855.0485798245613 < 1.159664368387921.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 35.932003148641606 (T) = (0 -844232.9663690476) / Math.Sqrt((1336600925.3741045 / (299)) + (233954988.83817908 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3192853156696506 = (1240215.5937029019 - 844232.9663690476) / 1240215.5937029019 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: None)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 67.67412769660895 < 114.41371973382003.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 162.587089781551 (T) = (0 -67612.41428831885) / Math.Sqrt((35972292.04954586 / (299)) + (7617.089525561898 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4586342013563767 = (124892.28994096015 - 67612.41428831885) / 124892.28994096015 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 65.447462890625 < 113.02816719459967.
IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 108.2469132919689 (T) = (0 -65048.392135474605) / Math.Sqrt((40260566.3773139 / (299)) + (318514.93523261254 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.47428257669276286 = (123732.61613865012 - 65048.392135474605) / 123732.61613865012 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: None)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 121.32318425760286 < 395.20932843749995.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 337.5119134548313 (T) = (0 -121427.27786385125) / Math.Sqrt((245575996.57733744 / (299)) + (21670.957723595282 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.7171604695854883 = (429315.08790831023 - 121427.27786385125) / 429315.08790831023 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 66.40484953843826 < 112.11424154777202.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 136.94030989538354 (T) = (0 -66204.23678888459) / Math.Sqrt((39798730.9798304 / (299)) + (80490.95056696214 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4627153357832368 = (123220.03808799393 - 66204.23678888459) / 123220.03808799393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 56.736213687241204 < 85.10100684226191.
IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 178.04260049386656 (T) = (0 -56712.29185267857) / Math.Sqrt((11050580.895957984 / (299)) + (1144.508337683549 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3781834113756635 = (91204.21180487462 - 56712.29185267857) / 91204.21180487462 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 123.66372039041994 < 391.93254499914144.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 295.4759381069475 (T) = (0 -123768.36156748436) / Math.Sqrt((316289795.1461345 / (299)) + (21899.551874854187 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.7116543279791785 = (429236.0648248158 - 123768.36156748436) / 429236.0648248158 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 431.5397913288288 < 612.2517528429058.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 72.40870562494796 (T) = (0 -434407.69508157717) / Math.Sqrt((36170091.94738617 / (299)) + (16449743.870055582 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.3250246352672673 = (643590.4445988009 - 434407.69508157717) / 643590.4445988009 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 57.635937977234164 < 85.70028075931343.
IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 136.3483433912403 (T) = (0 -57788.549343014376) / Math.Sqrt((12137244.648876978 / (299)) + (46580.45793060363 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.373570727256902 = (92250.71665307341 - 57788.549343014376) / 92250.71665307341 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 0c419b05455970ea4b009c14984785b59ba7b624 |
| Compare | da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 6.46 ms | 3.13 ms | 0.48 | 0.01 | False | |||||
| Count - Duration of single invocation | 5.18 ms | 3.02 ms | 0.58 | 0.00 | False | |||||
| Count - Duration of single invocation | 6.70 ms | 3.12 ms | 0.47 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: None)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.128152392708333 < 6.25696973922619.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 191.81679991897212 (T) = (0 -3122108.0431089746) / Math.Sqrt((108598117747.8105 / (299)) + (73068324.15853919 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5512400885359461 = (6957190.166392701 - 3122108.0431089746) / 6957190.166392701 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.0236688552083333 < 4.9209295256448415.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 630.6207756396386 (T) = (0 -3022415.5094494047) / Math.Sqrt((3108218991.9616356 / (299)) + (3141751.182847873 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.4191992145416341 = (5203876.415325653 - 3022415.5094494047) / 5203876.415325653 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.1205713917410716 < 6.311166585913771.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 180.53809915921443 (T) = (0 -3108912.255365728) / Math.Sqrt((94136531393.68825 / (299)) + (271870922.0377125 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5521585539008543 = (6941993.159510876 - 3108912.255365728) / 6941993.159510876 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 0c419b05455970ea4b009c14984785b59ba7b624 |
| Compare | da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| MatchesWord - Duration of single invocation | 1.53 μs | 967.06 ns | 0.63 | 0.00 | False | |||||
| CtorInvoke - Duration of single invocation | 205.09 μs | 182.97 μs | 0.89 | 0.02 | False | |||||
| CtorInvoke - Duration of single invocation | 218.97 μs | 195.06 μs | 0.89 | 0.01 | False | |||||
| ReplaceWords - Duration of single invocation | 2.56 μs | 2.35 μs | 0.92 | 0.01 | False | |||||
| ReplaceWords - Duration of single invocation | 2.52 μs | 2.23 μs | 0.88 | 0.00 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 967.0600495366923 < 1.4523714580605447.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 37.4098172216124 (T) = (0 -951.5491006253225) / Math.Sqrt((198.34352515138724 / (299)) + (481.17907226225026 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.379131719408099 = (1532.610265929786 - 951.5491006253225) / 1532.610265929786 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 182.97120947398187 < 196.3306870555556.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 4/25/2023 9:59:52 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 14.258326563329302 (T) = (0 -182410.51797095535) / Math.Sqrt((27449262.053110823 / (299)) + (24379429.386056934 / (26))) is greater than 1.967335607330539 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (26) - 2, .975) and 0.07348188188263198 = (196877.44298147468 - 182410.51797095535) / 196877.44298147468 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 195.05846055327865 < 210.0455501664844.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 9:59:52 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 41.48448139554595 (T) = (0 -194912.46608164132) / Math.Sqrt((19419862.041481767 / (299)) + (2871640.4071112033 / (26))) is greater than 1.967335607330539 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (26) - 2, .975) and 0.0818417260005535 = (212286.34713774722 - 194912.46608164132) / 212286.34713774722 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.351627338014289 < 2.409416184279373.
IsChangePoint: Marked as a change because one of 4/30/2023 6:17:41 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 24.78206896265696 (T) = (0 -2332.973430609114) / Math.Sqrt((960.78971805754 / (299)) + (530.0719888725904 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.0922649253653842 = (2570.103872595415 - 2332.973430609114) / 2570.103872595415 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.225991144873068 < 2.3835775649152118.
IsChangePoint: Marked as a change because one of 4/30/2023 6:17:41 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 32.79470743938113 (T) = (0 -2229.5750297051986) / Math.Sqrt((422.12640311557425 / (299)) + (430.60615955685626 / (6))) is greater than 1.967824097752068 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (6) - 2, .975) and 0.11176463651442674 = (2510.11738708083 - 2229.5750297051986) / 2510.11738708083 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 0c419b05455970ea4b009c14984785b59ba7b624 |
| Compare | da0aa0cb6944dd49d6c1d1859c4530fe7e38b76f |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 1.94 secs | 855.51 ms | 0.44 | 0.03 | False | |||||
| Count - Duration of single invocation | 2.00 secs | 867.49 ms | 0.43 | 0.02 | False | |||||
| Count - Duration of single invocation | 1.55 secs | 722.99 ms | 0.47 | 0.00 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 855.5071146666667 < 1.8428422200416668.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 212.34889358528096 (T) = (0 -856225257.5476191) / Math.Sqrt((9129647622444230 / (299)) + (1031458394925.2771 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5801748496725775 = (2039480619.2050364 - 856225257.5476191) / 2039480619.2050364 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, NonBacktracking)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 867.492054 < 1.860494975008333.
IsChangePoint: Marked as a change because one of 3/28/2023 3:45:48 AM, 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 192.8586946331901 (T) = (0 -864538031) / Math.Sqrt((8897856189767393 / (299)) + (17452503769058 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5805170614195236 = (2060961129.7317193 - 864538031) / 2060961129.7317193 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 722.9940353333334 < 1.4749771371916665.
IsChangePoint: Marked as a change because one of 5/1/2023 3:42:23 PM, 5/2/2023 10:35:24 AM falls between 4/23/2023 8:38:14 PM and 5/2/2023 10:35:24 AM.
IsImprovementStdDev: Marked as improvement because 622.9436841813911 (T) = (0 -724235239.3333334) / Math.Sqrt((72212197937473.47 / (299)) + (3081174739232 / (2))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (2) - 2, .975) and 0.5345040752417265 = (1555835831.8806338 - 724235239.3333334) / 1555835831.8806338 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository