Propagated headers get mixed up without previous async middleware

[orthoplex64]

Describe the bug

When using app.UseHeaderPropagation(); as the first middleware in the pipeline and sending many simultaneous requests, I've observed incorrect headers being propagated sometimes.
When inserting any async middleware before header propagation (e.g. app.Use(async (context, next) => await next.Invoke());), all propagated headers are correct.

To Reproduce

Steps to reproduce the behavior (ASP.NET Core 3.0 by default):

1. Clone this tiny demo project: https://github.com/orthoplex64/dotnet-demos
2. Execute dotnet run --project MangledHeaders in the cloned repository
3. Navigate to http://localhost:5000 and see all the wrong headers being returned
4. Uncomment .Use(async (context, next) => await next.Invoke()) in Startup.cs, re-run and see all the correct headers being returned
   Reproducible on ASP.NET Core 3.0 and 3.1. There is also a package reference to the ASP.NET Core 2.1/2.2 backport of the header propagation middleware, on which it is reproducible as well.

Expected behavior

The correct headers (i.e. the ones included in the current request) should be propagated on requests sent by HTTP Clients to which the header-propagating message handler has been added.

Screenshots

Additional context

I don't think it matters here, but the bug reporting template says to include the output of dotnet --info, so here it is:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.100-preview1-014459
 Commit:    ac3b59712d

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.100-preview1-014459\

Host (useful for support):
  Version: 3.1.0-preview1.19506.1
  Commit:  bbf5542781

.NET Core SDKs installed:
  2.2.100 [C:\Program Files\dotnet\sdk]
  2.2.203 [C:\Program Files\dotnet\sdk]
  3.0.100 [C:\Program Files\dotnet\sdk]
  3.1.100-preview1-014459 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0-preview1.19508.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0-preview1.19506.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.0-preview1.19506.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[benaadams]

@davidfowl related? #13991

[davidfowl]

Yes it’s the same bug

[jkotalik]

@davidfowl ping

[alefranz]

What's the plan for 3.1? is this bug significantly important to include the fix?

[alefranz]

Due to this subtle bug, it is risky to use the HeaderPropagation middleware at the moment.
Is there any chance we can include the workaround #15435 in a patch release?
The kestrel fix #14146 is still in progress and I doubt it will be backported to 3.1

If not possible to patch, should we document the bug somewhere (currently there is no documentation for this middleware) with some steps to take when using the middleware to avoid the bug? (e.g.: making sure this is not used as first async middleware in the pipeline)

/cc @rynowak