Runtime async should avoid boxing custom awaiters on suspension

[jakobbotsch]

Today for async1 the IL generated by Roslyn will save the awaiter into the display class as a normal field. The flow looks like:

1. IsCompleted is checked
2. If false, a copy of the awaiter value is copied into the display class
3. AsyncTaskMethodBuilder.AwaitUnsafeOnCompleted is called on a local
4. The call stack unwinds and continuations are queued. This requires synchronization since OnCompleted was already called, so the continuations may have already executed.

(As an aside, the fact that 2 and 3 do not happen on the same value is sort of strange, but presumably cannot be changed.)

For runtime async, however, the approach is different. It looks like:

1. IsCompleted is checked
2. If false immediately return. The call stack unwinds and continuations are created and linked. This does not require synchronization.
3. OnCompleted is called with the head continuation (from the deepest call frame)

This approach means that we need to communicate the awaiter back from step 2 to step 3. That's currently being done by boxing it and saving it in TLS. We should see if we can find a way to avoid this boxing step. The value should already be in the continuation since GetResult is called after the await, so we can reuse it from there. Step 3 will then require some stub to extract and call OnCompleted on the awaiter (particularly because it needs to be done on a copy to match async1's semantics).

cc @VSadov @agocke @333fred

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.