System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter failing on Windows Server 2022

[runfoapp]

System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter [FAIL]
    Assert.True() Failure
    Expected: True
    Actual:   False
    Stack Trace:
    /_/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs(191,0): at System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter()

Runfo Tracking Issue: system.text.tests.encodingextensionstests.convert_encoder_readonlyspan_ibufferwriter

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1619441	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1618950	runtime-extra-platforms	PR 65518	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1618727	runtime-extra-platforms	PR 65506	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1618267	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1617545	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1616876	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1615358	runtime-extra-platforms	PR 65120	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1614084	runtime-extra-platforms	PR 65035	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1612422	runtime-extra-platforms	PR 65120	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1611370	runtime-extra-platforms	PR 64452	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1610942	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1610175	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1609873	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1609271	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1608296	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1607081	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1606293	runtime-extra-platforms	PR 64952	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1605694	runtime-extra-platforms	PR 60905	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1603278	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1602281	runtime-extra-platforms	PR 65070	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1602227	runtime-extra-platforms	PR 64452	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1600670	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1599891	runtime-extra-platforms	PR 64330	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1599818	runtime-extra-platforms	PR 64567	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1599183	runtime-extra-platforms	Rolling	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1598876	runtime-extra-platforms	PR 60905	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py
1598024	runtime-extra-platforms	PR 64830	net7.0-windows-Release-x86-CoreCLR_release-Windows.Amd64.Server2022.Open	console.log			runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
5	17	27

Tagging subscribers to this area: @dotnet/area-system-text-encoding
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runfo Creating Tracking Issue (data being generated)

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Text.Encoding, untriaged
Milestone:	-

[elinor-fung]

Failed in today's rolling build 1599183
Passed in yesterday's rolling build 1597902

Diff: ebcd332...56fcb74

[elinor-fung]

The passing run was also on Server20H2, while the failing run was on Server2022.

@safern could this failure also be related to the ICU differences in the environment? Asking because this only seems to be on x86 and I saw your comment about x86 in #64827.

There are a couple others that seem encoding-related and are failing for x86 on Windows Server 2022 too:

• System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers failing in runtime-extra-platforms #65019
• System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync failing in runtime-extra-platforms #65021

[safern]

@safern could this failure also be related to the ICU differences in the environment? Asking because this only seems to be on x86 and I saw your comment about x86 in #64827.

I wouldn't think that'd be the case because ICU is being loaded correctly (the ICU tests are not failing), and if there was a change in the ICU data for globalization, I guess we would see these fail on x64 (I just double checked and we do run x64 on this platform as well).

Adding @tarekgh in case he has more insight.

[tarekgh]

@elinor-fung @safern the encoding is independent of the ICU. This failure is not related to ICU. @GrabYourPitchforks may guess what could be the problem here.

The failing line is

runtime/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs

Line 191 in 205f70e

Assert.True(

            Assert.True(
                Encoding.UTF8.GetBytes("Hello" + new string('x', 20_000_000) + "\U00010000" + new string('x', 20_000_000) + '\ufffd').AsSpan().SequenceEqual(writer.WrittenSpan));

[danmoseley]

This is failing always and only on windows.amd64.server2022.open since the queue was enabled, and passing everywhere else.

[elinor-fung]

The Decoder parallel of this test (Convert_Decoder_ReadOnlySpan_IBufferWriter) seems to be failing a decent amount as well.

Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-79f7f0b90e35498294/System.Memory.Tests/1/console.7a63d464.log?sv=2019-07-07&se=2022-03-01T10%3A04%3A45Z&sr=c&sp=rl&sig=v5h6CduPBAk1q3AoRa82VsV%2FWpG56mp4MHD0Y%2BVlmPQ%3D

Assert.Equal() Failure
                                 ↓ (pos 10663021)
Expected: ···xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx···
Actual:   ···xxxxxxxxxxxxxxxxxxxx\0\0\0\0\0\0\0\0xxxxxxxx\0\0\0\0\0\0\0\0xxxxxxxx\0\0\0\0\0\0\0\0x···
                                 ↑ (pos 10663021)

at System.Text.Tests.EncodingExtensionsTests.Convert_Decoder_ReadOnlySpan_IBufferWriter() in /_/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs:line 98

runtime/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs

Lines 98 to 100 in 205f70e

	Assert.Equal(
	expected: "Hello" + new string('x', 20_000_000) + '\u0800' + new string('x', 20_000_000) + '\ufffd',
	actual: writer.WrittenSpan.ToString());

[elinor-fung]

This is hitting consistently on Server 2022. We will need to update the main runtime pipeline to use a Server 2022 queue soon. @GrabYourPitchforks any ideas what might be going on here?

[mdh1418]

Also hit this in an unrelated PR #64952
Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1606293&view=logs&j=457f7e88-dfa2-5bd9-f871-fdf124c2477d&s=96ac2280-8cb4-5df5-99de-dd2da759617d
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-64952-merge-bb663c35c087459bb8/System.Memory.Tests/1/console.ca93e470.log?sv=2019-07-07&se=2022-03-03T01%3A58%3A49Z&sr=c&sp=rl&sig=8twS3bwRjJKf0aJthzMSrl8XnyO1gthfCqRNFarRKm0%3D

    System.Text.Tests.EncodingExtensionsTests.Convert_Decoder_ReadOnlySpan_IBufferWriter [FAIL]
      Assert.Equal() Failure
                                       ↓ (pos 13968685)
      Expected: ···xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx···
      Actual:   ···xxxxxxxxxxxxxxxxxxxx\0\0\0\0\0\0\0\0xxxxxxxx\0\0\0\0\0\0\0\0xxxxxxxx\0\0\0\0\0\0\0\0x···
                                       ↑ (pos 13968685)
      Stack Trace:
        /_/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs(98,0): at System.Text.Tests.EncodingExtensionsTests.Convert_Decoder_ReadOnlySpan_IBufferWriter()
    System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Memory/tests/EncodingExtensions/EncodingExtensionsTests.cs(191,0): at System.Text.Tests.EncodingExtensionsTests.Convert_Encoder_ReadOnlySpan_IBufferWriter()
  Finished:    System.Memory.Tests

[danmoseley]

Noting these are different machines (anyway, bad memory usually looks like a bit flip). Quite curious it's always Server 2022