Peculiar string operation performance

[Suchiman]

So i got nerd sniped on IRC with how fast you can append characters to create a string.
I created the following benchmark to measure a few alternatives just before i got told it's for an old MVC4 app. Then I've noticed that the performance is consistently worse on .NET Core, in particular for StringBuilder and even more so if you supply a capacity for StringBuilder. I then suspected Tiered Compilation but i couldn't figure how to really disable it for BDN.NET, <TieredCompilation>false</TieredCompilation> had at best an effect on the InProcess Benchmark. Also it appears that BDN.NET or the API it is using for measuring the Allocated number is incorrect on .NET Core, the .NET Framework numbers seem more plausible (allocating an char[] of 50k elements and a string from that char[] certainly doesn't allocate exactly as much as just allocating the string).

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text;

namespace ConsoleApp37
{
    class Program
    {
        static void Main(string[] args)
        {
            BenchmarkRunner.Run<ConcatRace>();
        }
    }

    //[InProcess]
    [MemoryDiagnoser]
    public class ConcatRace
    {
        const int length = 50000;

        [Benchmark]
        public string StringBuilderAddChar()
        {
            var result = new StringBuilder();

            for (int i = 0; i < length; i++)
            {
                result.Append('s');
            }

            return result.ToString();
        }

        [Benchmark]
        public string StringBuilderAddCharOptimized()
        {
            var result = new StringBuilder(length);

            for (int i = 0; i < length; i++)
            {
                result.Append('s');
            }

            return result.ToString();
        }

        [Benchmark]
        public unsafe string PointerAddChar()
        {
            var result = new char[length];

            fixed (char* fixedPointer = result)
            {
                var pointer = fixedPointer;
                for (int i = 0; i < length; i++)
                {
                    *pointer++ = 's';
                }
            }

            return new string(result);
        }

        [Benchmark]
        public unsafe string NormalAddChar()
        {
            var result = new char[length];

            for (int i = 0; i < result.Length; i++)
            {
                result[i] = 's';
            }

            return new string(result);
        }

        [Benchmark]
        public unsafe string FastPointerAddChar()
        {
            var result = new string('\0', length);

            fixed (char* fixedPointer = result)
            {
                var pointer = fixedPointer;
                for (int i = 0; i < length; i++)
                {
                    *pointer++ = 's';
                }
            }

            return result;
        }
    }
}

.NET Framework 4.7.2

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362
Intel Core i5-4670K CPU 3.40GHz (Haswell), 1 CPU, 4 logical and 4 physical cores
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.8.3815.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.8.3815.0

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
StringBuilderAddChar	109.43 us	0.3330 us	0.2952 us	62.3779	31.1279	31.1279	209.02 KB
StringBuilderAddCharOptimized	100.69 us	0.5440 us	0.5089 us	62.3779	62.3779	62.3779	195.87 KB
PointerAddChar	33.79 us	0.2182 us	0.1934 us	62.4390	62.4390	62.4390	195.37 KB
NormalAddChar	38.40 us	0.1843 us	0.1539 us	62.4390	62.4390	62.4390	195.37 KB
FastPointerAddChar	24.62 us	0.2241 us	0.2096 us	31.2195	31.2195	31.2195	97.69 KB

.NET Core 3.0 RC1 (OutOfProcess)

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362
Intel Core i5-4670K CPU 3.40GHz (Haswell), 1 CPU, 4 logical and 4 physical cores
.NET Core SDK=3.0.100-rc1-014190
  [Host]     : .NET Core 3.0.0-rc1-19456-20 (CoreCLR 4.700.19.45506, CoreFX 4.700.19.45604), 64bit RyuJIT
  DefaultJob : .NET Core 3.0.0-rc1-19456-20 (CoreCLR 4.700.19.45506, CoreFX 4.700.19.45604), 64bit RyuJIT

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
StringBuilderAddChar	120.80 us	0.5277 us	0.4936 us	62.2559	31.0059	31.0059	208.55 KB
StringBuilderAddCharOptimized	142.44 us	1.1631 us	1.0879 us	62.2559	62.2559	62.2559	97.73 KB
PointerAddChar	35.33 us	0.2321 us	0.2057 us	62.4390	62.4390	62.4390	97.68 KB
NormalAddChar	40.27 us	0.4038 us	0.3777 us	62.4390	62.4390	62.4390	97.68 KB
FastPointerAddChar	29.78 us	0.1331 us	0.1245 us	31.2195	31.2195	31.2195	97.68 KB

.NET Core 3.0 RC1 (InProcess)

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362
Intel Core i5-4670K CPU 3.40GHz (Haswell), 1 CPU, 4 logical and 4 physical cores
.NET Core SDK=3.0.100-rc1-014190
  [Host] : .NET Core 3.0.0-rc1-19456-20 (CoreCLR 4.700.19.45506, CoreFX 4.700.19.45604), 64bit RyuJIT

Job=InProcess  Toolchain=InProcessEmitToolchain  

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
StringBuilderAddChar	121.68 us	0.4015 us	0.3352 us	62.3779	31.1279	31.1279	208.55 KB
StringBuilderAddCharOptimized	111.83 us	1.4062 us	1.3154 us	62.3779	62.3779	62.3779	97.73 KB
PointerAddChar	43.64 us	0.5115 us	0.4784 us	62.4390	62.4390	62.4390	97.68 KB
NormalAddChar	48.63 us	0.6182 us	0.5782 us	62.4390	62.4390	62.4390	97.68 KB
FastPointerAddChar	29.98 us	0.3981 us	0.3724 us	31.2195	31.2195	31.2195	97.68 KB

[jkotas]

cc @adamsitnik

[danmoseley]

Also reported here https://github.com/dotnet/coreclr/issues/25774

[danmoseley]

Seems a duplicate of #13107. Lets continue discussion there.

[Suchiman]

@danmosemsft imo this is a superset of that other ticket which is exclusive to stringbuilder

[danmoseley]

@Suchiman I see your observations about non StringBuilder appends but it's not clear to me that there's a problem with those?

[Suchiman]

They're consistently slower even if only by a small margin (between 5% and 18%) and then there's the incorrect measurement of Allocated

[danmoseley]

@Suchiman you're right -- I had read the tables in reverse.

[adamsitnik]

I am going to investigate it tomorrow

[adamsitnik]

Hi @Suchiman

Also it appears that BDN.NET or the API it is using for measuring the Allocated number is incorrect on .NET Core

You are right. Before .NET Core 3.0 preview 6+ and BenchmarkDotNet 0.11.6, BenchmarkDotNet was using GC.GetAllocatedBytesForCurrentThread to get the number of allocated bytes. Recently we have switched to the new API added recently to .NET Core 3.0 - GC.GetTotalAllocatedBytes dotnet/BenchmarkDotNet#1155 and this solves the problem of missing allocations.

Using the latest version of BDN from our CI feed and .NET Core 3.0 I get the correct number of allocated bytes:

BenchmarkDotNet=v0.11.5.1159-nightly, OS=Windows 10.0.18362
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-alpha1-013817
[Host] : .NET Core 3.0.0 (CoreCLR 4.700.19.46205, CoreFX 4.700.19.46214), X64 RyuJIT
Job-XMGAOR : .NET Framework 4.8 (4.8.4010.0), X64 RyuJIT
Job-JHKCBR : .NET Core 3.0.0 (CoreCLR 4.700.19.46205, CoreFX 4.700.19.46214), X64 RyuJIT

Method	Runtime	Toolchain	Mean	Ratio	Allocated
StringBuilderAddChar	.NET 4.7.2	net472	124.16 us	1.00	208.77 KB
StringBuilderAddChar	.NET Core 3.0	netcoreapp3.0	147.46 us	1.21	208.56 KB
StringBuilderAddCharOptimized	.NET 4.7.2	net472	120.43 us	1.00	195.87 KB
StringBuilderAddCharOptimized	.NET Core 3.0	netcoreapp3.0	140.72 us	1.17	195.41 KB
PointerAddChar	.NET 4.7.2	net472	41.11 us	1.00	195.37 KB
PointerAddChar	.NET Core 3.0	netcoreapp3.0	40.19 us	0.98	195.36 KB
NormalAddChar	.NET 4.7.2	net472	42.01 us	1.00	195.37 KB
NormalAddChar	.NET Core 3.0	netcoreapp3.0	44.21 us	1.05	195.36 KB
FastPointerAddChar	.NET 4.7.2	net472	29.44 us	1.00	97.69 KB
FastPointerAddChar	.NET Core 3.0	netcoreapp3.0	32.82 us	1.12	97.68 KB

i couldn't figure how to really disable it for BDN.NET

The benchmark should get promoted to Tier 1 during Warmup or Pilot stage, this is why BDN does not expose a possibility to disable Tiered JIT. However, you can achieve that by setting environment variables

[adamsitnik]

When it comes to the most important aspect - for your benchmark .NET Core 3.0 is slower than .NET 4.8. I am going to investigate that.

[adamsitnik]

Also, I wanted to mention that we track the performance of StringBuilder in dotnet/performance repository: https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/corefx/System.Runtime/Perf.StringBuilder.cs

I've run all the benchmarks we have for 4.8 vs 3.0 and in general .NET Core 3.0 is slower than 4.8 only in a single case:

Method	Runtime	Toolchain	length	Mean	Ratio	Allocated
ctor	.NET 4.7.2	net472	?	15.09 ns	1.00	104 B
ctor	.NET Core 3.0	netcoreapp3.0	?	12.98 ns	0.88	104 B
StringConcat	.NET 4.7.2	net472	?	45,430,416.67 ns	1.00	275971747 B
StringConcat	.NET Core 3.0	netcoreapp3.0	?	48,247,462.50 ns	1.06	275701266 B
StringBuilderAppend	.NET 4.7.2	net472	?	129,789.88 ns	1.00	565002 B
StringBuilderAppend	.NET Core 3.0	netcoreapp3.0	?	126,277.46 ns	0.97	563552 B
Append_ValueTypes	.NET 4.7.2	net472	?	7,435,766.25 ns	1.00	2225087 B
Append_ValueTypes	.NET Core 3.0	netcoreapp3.0	?	2,285,843.80 ns	0.31	932616 B
StringBuilderToString	.NET 4.7.2	net472	?	79,553.35 ns	1.00	550032 B
StringBuilderToString	.NET Core 3.0	netcoreapp3.0	?	80,571.22 ns	1.01	550024 B
AppendMemory	.NET 4.7.2	net472	?	41,637.46 ns	1.00	235515 B
AppendMemory	.NET Core 3.0	netcoreapp3.0	?	29,821.27 ns	0.72	203040 B
Append	.NET 4.7.2	net472	0	26,470.82 ns	1.00	105 B
Append	.NET Core 3.0	netcoreapp3.0	0	31,749.11 ns	1.20	104 B
ctor_string	.NET 4.7.2	net472	100	32.81 ns	1.00	273 B
ctor_string	.NET Core 3.0	netcoreapp3.0	100	32.89 ns	1.01	272 B
Append	.NET 4.7.2	net472	200	1,068,612.78 ns	1.00	4035648 B
Append	.NET Core 3.0	netcoreapp3.0	200	926,691.15 ns	0.87	4028106 B
ctor_string	.NET 4.7.2	net472	1000	200.28 ns	1.00	2078 B
ctor_string	.NET Core 3.0	netcoreapp3.0	1000	197.00 ns	0.98	2072 B

[Suchiman]

BenchmarkDotNet was using GC.GetAllocatedBytesForCurrentThread to get the number of allocated bytes

Okay, but there's only a single thread involved here, why is this API not delivering the correct result?