[Perf] Regressions in System.Collections.TryGetValueFalse<String, String>

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	59c592cc8d2778bcc6173baa2b25b13190e42990
Compare	6bfc5f21dea7b550f1c807454d45408ef34764e1
Diff	Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IDictionary	9.34 μs	11.07 μs	1.19
Dictionary	8.17 μs	9.92 μs	1.21

![graph]
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.TryGetValueFalse<String, String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.IDictionary(Size: 512)

System.Collections.TryGetValueFalse<String, String>.Dictionary(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

category:performance
theme:benchmarks

Tagging subscribers to this area: @eiriktsarpalis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	59c592cc8d2778bcc6173baa2b25b13190e42990
Compare	6bfc5f21dea7b550f1c807454d45408ef34764e1
Diff	Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IDictionary	9.34 μs	11.07 μs	1.19
Dictionary	8.17 μs	9.92 μs	1.21

![graph]
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.TryGetValueFalse<String, String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.IDictionary(Size: 512)

System.Collections.TryGetValueFalse<String, String>.Dictionary(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	DrewScoggins
Assignees:	AndyAyersMS
Labels:	arch-x64, area-System.Collections, os-linux, tenet-performance, tenet-performance-benchmarks, untriaged
Milestone:	-

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	59c592cc8d2778bcc6173baa2b25b13190e42990
Compare	6bfc5f21dea7b550f1c807454d45408ef34764e1
Diff	Diff

Improvemnts in System.Collections.ContainsKeyFalse<String, String>

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IDictionary	11.48 μs	9.45 μs	0.82
Dictionary	9.95 μs	8.25 μs	0.83

![graph]
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.ContainsKeyFalse<String, String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.IDictionary(Size: 512)

System.Collections.ContainsKeyFalse<String, String>.Dictionary(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

Looking at the full test trends for TryGetValue and ContainsKeyFalse, seems to show that when one gets faster the other get slower.

[AndyAyersMS]

@DrewScoggins the zip archives linked above lose all file permissions (or else I'm doing it wrong). Makes it painful to download and then run the binaries.

andy@andy-ubuntu$ unzip -Zl 6d58d995-201b-444b-a499-a528190087f3.zip Core_Root/corerun 
?---------  2.0 unx   108912 b-    39058 defN 21-Apr-05 14:49 Core_Root/corerun

after an unzip:

---------- 1 andy andy   108912 Apr  5 14:49 corerun

Can we instead get a zipped up tar archive?

[AndyAyersMS]

FWIW I can't repro the above regression with local builds on my unix box, which is why I wanted to grab the exact bits used by the runs.

[DrewScoggins]

Those are the zips that we send to Helix, and we don't really have another place where we can easily get them. If I remember the only thing you had to chmod +x was corerun, and then everything worked. Maybe we can look at including a little shell script that setups the binaries for repro?

[AndyAyersMS]

If this is what you send, I wonder what helix does to work around this?

Let me see if fixing corerun to be +x and all the others +r does the trick.

[AndyAyersMS]

It requires more futzing about than just that, not quite sure what BDN is doing... at any rate I have the same non-result with the downloaded builds as I did with the local builds. Could be my ancient HW I suppose.

TryGetValueFalse

Base, Download

BenchmarkDotNet=v0.12.1.1521-nightly, OS=ubuntu 18.04
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-preview.3.21202.5
  [Host]     : .NET 6.0.0 (6.0.21.20104), X64 RyuJIT
  Job-CXOBIO : .NET 6.0.0 (6.0.21.20502), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  Toolchain=CoreRun  
IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15  
WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max
Dictionary	512	14.85 µs	0.060 µs	0.047 µs	14.84 µs	14.79 µs	14.93 µs
IDictionary	512	16.57 µs	0.105 µs	0.093 µs	16.58 µs	16.41 µs	16.71 µs
SortedList	512	420.46 µs	1.288 µs	1.142 µs	420.31 µs	418.88 µs	422.94 µs
SortedDictionary	512	460.61 µs	17.330 µs	19.957 µs	447.18 µs	443.50 µs	487.07 µs
ConcurrentDictionary	512	22.84 µs	0.076 µs	0.068 µs	22.81 µs	22.77 µs	23.00 µs
ImmutableDictionary	512	37.58 µs	0.208 µs	0.184 µs	37.57 µs	37.33 µs	37.91 µs
ImmutableSortedDictionary	512	422.83 µs	1.118 µs	0.933 µs	422.94 µs	421.12 µs	424.30 µs

Base, Local

BenchmarkDotNet=v0.12.1.1521-nightly, OS=ubuntu 18.04
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-preview.3.21202.5
  [Host]     : .NET 6.0.0 (6.0.21.20104), X64 RyuJIT
  Job-NDBUIM : .NET 6.0.0 (42.42.42.42424), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  Toolchain=CoreRun  
IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15  
WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max
Dictionary	512	14.88 µs	0.070 µs	0.062 µs	14.86 µs	14.80 µs	15.01 µs
IDictionary	512	16.67 µs	0.202 µs	0.169 µs	16.61 µs	16.53 µs	17.13 µs
SortedList	512	417.57 µs	1.038 µs	0.867 µs	417.49 µs	416.20 µs	418.93 µs
SortedDictionary	512	438.12 µs	3.399 µs	3.013 µs	438.20 µs	433.96 µs	443.15 µs
ConcurrentDictionary	512	22.92 µs	0.053 µs	0.044 µs	22.90 µs	22.86 µs	22.99 µs
ImmutableDictionary	512	36.94 µs	0.134 µs	0.126 µs	36.92 µs	36.75 µs	37.18 µs
ImmutableSortedDictionary	512	422.31 µs	0.916 µs	0.765 µs	422.20 µs	420.92 µs	424.09 µs

Diff, Download

BenchmarkDotNet=v0.12.1.1521-nightly, OS=ubuntu 18.04
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-preview.3.21202.5
  [Host]     : .NET 6.0.0 (6.0.21.20104), X64 RyuJIT
  Job-PLJKLO : .NET 6.0.0 (6.0.21.20602), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  Toolchain=CoreRun  
IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15  
WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max
Dictionary	512	14.94 µs	0.108 µs	0.101 µs	14.92 µs	14.82 µs	15.18 µs
IDictionary	512	16.73 µs	0.142 µs	0.126 µs	16.78 µs	16.49 µs	16.90 µs
SortedList	512	434.83 µs	1.179 µs	0.985 µs	434.84 µs	433.51 µs	436.95 µs
SortedDictionary	512	468.35 µs	1.755 µs	1.465 µs	468.34 µs	466.65 µs	471.78 µs
ConcurrentDictionary	512	23.12 µs	0.299 µs	0.280 µs	22.98 µs	22.84 µs	23.75 µs
ImmutableDictionary	512	37.81 µs	0.702 µs	0.548 µs	37.64 µs	37.23 µs	39.04 µs
ImmutableSortedDictionary	512	445.27 µs	5.198 µs	4.608 µs	442.97 µs	441.13 µs	454.60 µs

Diff, Local

BenchmarkDotNet=v0.12.1.1521-nightly, OS=ubuntu 18.04
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-preview.3.21202.5
  [Host]     : .NET 6.0.0 (6.0.21.20104), X64 RyuJIT
  Job-KXUFTT : .NET 6.0.0 (42.42.42.42424), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable  Toolchain=CoreRun  
IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15  
WarmupCount=1  

Method	Size	Mean	Error	StdDev	Median	Min	Max
Dictionary	512	14.74 µs	0.070 µs	0.062 µs	14.73 µs	14.64 µs	14.86 µs
IDictionary	512	16.36 µs	0.080 µs	0.075 µs	16.37 µs	16.27 µs	16.50 µs
SortedList	512	408.75 µs	3.088 µs	2.411 µs	407.43 µs	406.70 µs	413.31 µs
SortedDictionary	512	448.91 µs	1.650 µs	1.462 µs	449.24 µs	446.71 µs	451.09 µs
ConcurrentDictionary	512	22.67 µs	0.057 µs	0.048 µs	22.68 µs	22.58 µs	22.75 µs
ImmutableDictionary	512	37.48 µs	0.114 µs	0.101 µs	37.46 µs	37.31 µs	37.68 µs
ImmutableSortedDictionary	512	433.14 µs	3.722 µs	2.906 µs	433.14 µs	429.73 µs	436.68 µs