SequenceEqual broken for concurrent use?

[stephentoub]

Repro:

using System;
using System.Security.Cryptography;
using System.Threading;

class Program
{
    static void Main()
    {
        static void Work(object _)
        {
            while (true)
            {
                Span<byte> buffer1 = RandomNumberGenerator.GetBytes(1024 * 1024);
                Span<byte> buffer2 = new byte[buffer1.Length];
                buffer1.CopyTo(buffer2);

                if (!buffer1.SequenceEqual(buffer2)) // should never be false...
                {
                    while (!buffer1.SequenceEqual(buffer2)) ; // but if it is, it certainly shouldn't go back to true...
                    throw new Exception("What in the world just happened?!");
                }
            }
        }

        ThreadPool.QueueUserWorkItem(Work);
        ThreadPool.QueueUserWorkItem(Work);
        Console.ReadLine();
    }
}

This runs without error on .NET 5.

On master on my Windows machine (I've not tried other OSes), within a few seconds I get a failure like this:

Unhandled exception. System.Exception: SequenceEquals changed from false to true?!
   at Program.Work() in C:\Users\stoub\source\repos\ConsoleApp17\ConsoleApp17\Program.cs:line 25
   at Program.<>c.<Main>b__0_0(Object _) in C:\Users\stoub\source\repos\ConsoleApp17\ConsoleApp17\Program.cs:line 10
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() in D:\repos\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPool.cs:line 977
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in D:\repos\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPool.cs:line 705
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in D:\repos\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.cs:line 56
   at System.Threading.Thread.StartCallback() in D:\repos\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 105

Note that if I remove one of the ThreadPool calls so that there's no concurrency, it's never failed for me; only once I introduce the parallelism does it start to fail.

JIT bug? GC? Some kind of latent issue in SequenceEquals?

cc: @geoffkizer, @jkotas

[stephentoub]

As a follow-up, this still fails if I pin both buffers around the whole copy+compare operation. And it still fails if I move one of the allocations out of the loop. But it stops reproing for me if I move both allocations out of the loop. So maybe something to do with alignment?

[jkotas]

This looks like a thread suspension bug to me. @VSadov Could you please take a look since you have done refactoring of that code recently?

[VSadov]

Will take a look.

[VSadov]

Hmm, does not compile for me.

RandomNumberGenerator.GetBytes(1024 * 1024);

Is that a new API in 6.0 ?

[jkotas]

Here is a simpler repro that does not use that API:

using System;
using System.Threading;

class Program
{
    static object o = null;

    static void Main()
    {
        static void Work(object _)
        {
            while (true)
            {
                Span<byte> buffer1 = new byte[1024 * 1024];
                for (int i = 0; i < buffer1.Length; i++) buffer1[i] = (byte)i;
                Span<byte> buffer2 = new byte[buffer1.Length];
                buffer1.CopyTo(buffer2);

                if (!buffer1.SequenceEqual(buffer2)) // should never be false...
                {
                    o.ToString(); // AV to break into debugger
                    throw new Exception("What in the world just happened?!");
                }
            }
        }

        ThreadPool.QueueUserWorkItem(Work);
        ThreadPool.QueueUserWorkItem(Work);
        Thread.Sleep(-1);
    }
}

[VSadov]

That builds, but so far I do not see failures. Tried more tasks, and server GC as well.
(I am using clr that I've just built from the current master ~ few commits)

How long does it normally take to see a repro?

[stephentoub]

Seconds on my machine, using src from master this afternoon.

[jkotas]

Does you machine have AVX2? Also, are you using checked or release coreclr build?