[Perf -23%] System.Collections.IndexerSet<String>.Span

[DrewScoggins]

I have added a new feature to the performance issues. If you click on the name of the benchmark in the table, you will be taken to a page that shows the entire test history for that benchmark.

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Collections.IndexerSet

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
Span	338.68 ns	420.03 ns	1.24		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.IndexerSet<String>*'

Histogram

System.Collections.IndexerSet.Span(Size: 512)

[333.353 ; 347.615) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[347.615 ; 366.299) | @@@@@@@@@@@@@@
[366.299 ; 386.978) | @@@@@@@
[386.978 ; 401.231) | @@@@@
[401.231 ; 415.994) | 
[415.994 ; 430.257) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[430.257 ; 443.483) | @

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @eiriktsarpalis
See info in area-owners.md if you want to be subscribed.

[danmoseley]

@DrewScoggins the diff link is bad again. It covers Aug 4-Aug 14th and won't open.

The correct link, based on zooming into the graph, is seems to be https://www.github.com/dotnet/runtime/compare/55ed5cda016f1dd3d730e81165c40d45327e1ba4...d1c7946bb0892005a58ec647e3a6d8f3514cdc6b

Is it possible to tighten the diff before opening such issues? Maybe a script needs adjustment, or it has to be done manually.

[DrewScoggins]

I updated it. This will likely always be a manual process, at least for the time being.

[danmoseley]

Thank you. Can you pleaes also fix the typo in the repro instructions that I mentioned in an earlier issue? The Windows repro instructions should be like this I think

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter "System.Collections.IndexerSet<String>*"

[danmoseley]

Looking at the changes, there's no smoking gun but this one seems least unlikely to be relevant:

d1c7946 #40355

@trylek do you think that is possible?

Here is the test (parameterized for 'string'):
https://github.com/dotnet/performance/blob/8f00082e5f1ab8b86a98bf3bfc9c307a171912a3/src/benchmarks/micro/libraries/System.Collections/Indexer/IndexerSet.cs#L56-L62

[danmoseley]

@trylek could you help us with the question above?

[trylek]

Thanks Dan for the heads-up. Well, I certainly cannot rule it out. While the change was intended exactly to fix perf issues observed in generic algorithms after Fadi's introduction of expanding dictionaries, I can theoretically imagine corner cases where the additional amount of runtime work upon dictionary expansion and / or the counterpart JIT change to stop treating the dictionary pointer as invariant (and eligible for common subexpression elimination) can actually cause a perf regression. I'll investigate it in more detail tomorrow, I'll diff the assembly with and without my change and I'll try to repro the perf difference locally; I'll continue updating this thread with my findings.

[danmoseley]

Thank you @trylek