System.Net.Http.WinHttpResponseStream leading to crash in SafeWinHttpHandle.ReleaseHandle

[tdevere]

We are reporting daily crashes after moving our application from .Net 4.6 to .Net Core 2.0. When the crash occurs all 64 Cores are spiked at 100% CPU.

Example Call Stack:

dotnet/corefx#1: SafeWinHttpHandle.ReleaseHandle()

00 coreclr!EEPolicy::HandleFatalError
01 coreclr!ProcessCLRException
02 ntdll!RtlpExecuteHandlerForException
03 ntdll!RtlDispatchException
04 ntdll!KiUserExceptionDispatch
05 crypt32!ReleaseContextElement
06 crypt32!CertFreeCertificateContext
07 winhttp!WEBIO_REQUEST::{dtor}
08 winhttp!WEBIO_REQUEST::`scalar deleting destructor'
09 winhttp!HTTP_BASE_OBJECT::Dereference
0a winhttp!HTTP_USER_REQUEST::_SafeDetachSysReq
0b winhttp!HTTP_USER_REQUEST::Shutdown
0c winhttp!HTTP_REQUEST_HANDLE_OBJECT::SafeShutdownUsrReq
0d winhttp!_InternetCloseHandle
0e winhttp!WinHttpCloseHandle
       Child SP               IP Call Site
00000109e165da28 00007ffaf35a4f86 [InlinedCallFrame: 00000109e165da28] Interop+WinHttp.WinHttpCloseHandle(IntPtr)
00000109e165da28 00007ffa943ab463 [InlinedCallFrame: 00000109e165da28] Interop+WinHttp.WinHttpCloseHandle(IntPtr)
00000109e165dab0 00007ffaf1bb6b08 Interop+WinHttp+SafeWinHttpHandle.ReleaseHandle() [E:\A\_work\774\s\corefx\src\Common\src\Interop\Windows\winhttp\Interop.SafeWinHttpHandle.cs @ 59]
00000109e165dcd0 00007ffaf3482d33 [GCFrame: 00000109e165dcd0]
00000109e165dd08 00007ffaf3482d33 [GCFrame: 00000109e165dd08]
00000109e165de58 00007ffaf3482d33 [HelperMethodFrame_1OBJ: 00000109e165de58] System.Runtime.InteropServices.SafeHandle.InternalDispose()
00000109e165dfc0 00007ffaf1bd1ca7 System.Net.Http.WinHttpResponseStream.Dispose(Boolean) [E:\A\_work\774\s\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpResponseStream.cs @ 282]
00000109e165e000 00007ffaf2e1a72a System.IO.Stream.Close() [E:\A\_work\308\s\src\mscorlib\src\System\IO\Stream.cs @ 263]
00000109e165e030 00007ffaf1bc97d0 System.Net.Http.NoWriteNoSeekStreamContent+c.b__4_0(System.Threading.Tasks.Task, System.Object) [E:\A\_work\774\s\corefx\src\Common\src\System\Net\Http\NoWriteNoSeekStreamContent.cs @ 51]
00000109e165e070 00007ffaf2d871ce System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\308\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 145]
00000109e165e0e0 00007ffaf2e143d6 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2454]
00000109e165e180 00007ffaf2f78446 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\ThreadPoolTaskScheduler.cs @ 76]
00000109e165e1d0 00007ffaf2e439b3 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\TaskScheduler.cs @ 210]
00000109e165e230 00007ffaf2e800df System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\TaskContinuation.cs @ 256]
00000109e165e280 00007ffaf2e155af System.Threading.Tasks.Task.RunContinuations(System.Object) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 3263]
00000109e165e370 00007ffaf2e81705 System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\future.cs @ 425]
00000109e165e3b0 00007ffaf2e5dc89 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult) [E:\A\_work\308\s\src\mscorlib\src\System\Runtime\CompilerServices\AsyncMethodBuilder.cs @ 605]
00000109e165e3f0 00007ffaf2e5dc1c System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetResult(System.Threading.Tasks.Task`1) [E:\A\_work\308\s\src\mscorlib\src\System\Runtime\CompilerServices\AsyncMethodBuilder.cs @ 646]
00000109e165e420 00007ffaf1bd2532 System.Net.Http.WinHttpResponseStream+d__18.MoveNext() [E:\A\_work\774\s\corefx\src\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpResponseStream.cs @ 163]
00000109e165e4e0 00007ffaf2d871ce System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\308\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 145]
00000109e165e550 00007ffaf2d871ce System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\308\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 145]
00000109e165e5c0 00007ffaf2e143d6 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\308\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2454]
00000109e165e660 00007ffaf2e537f9 System.Threading.ThreadPoolWorkQueue.Dispatch() [E:\A\_work\308\s\src\mscorlib\src\System\Threading\ThreadPool.cs @ 582]

dotnet/corefx#2: SafeWinHttpHandle.ReleaseHandle()

       Child SP               IP Call Site
000000bb26d9c590 00007ffd81220c8a [FaultingExceptionFrame: 000000bb26d9c590]
000000bb26d9ca90 00007ffd6754e6c6 System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr, System.Net.Http.WinHttpRequestState, UInt32, IntPtr, UInt32)
000000bb26d9ea70 00007ffd68903190 [FaultingExceptionFrame: 000000bb26d9ea70]
000000bb26d9ef70 00007ffd6754e57e System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr, System.Net.Http.WinHttpRequestState, UInt32, IntPtr, UInt32)
000000bb26d9efe0 00007ffd6754e4a7 System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr, IntPtr, UInt32, IntPtr, UInt32)
000000bb26d9f030 00007ffd099b1332 DomainBoundILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int32, Int64, Int32)
000000bb26d9f2c8 00007ffd68a02e4a [InlinedCallFrame: 000000bb26d9f2c8] Interop+WinHttp.WinHttpCloseHandle(IntPtr)
000000bb26d9f2c8 00007ffd099b34c3 [InlinedCallFrame: 000000bb26d9f2c8] Interop+WinHttp.WinHttpCloseHandle(IntPtr)
000000bb26d9f350 00007ffd67536b08 Interop+WinHttp+SafeWinHttpHandle.ReleaseHandle()
000000bb26d9f4a0 00007ffd68a02d33 [GCFrame: 000000bb26d9f4a0]
000000bb26d9f618 00007ffd68a02d33 [GCFrame: 000000bb26d9f618]
000000bb26d9f6b8 00007ffd68a02d33 [HelperMethodFrame_1OBJ: 000000bb26d9f6b8] System.Runtime.InteropServices.SafeHandle.InternalFinalize()
000000bb26d9f7c0 00007ffd68359b16 System.Runtime.InteropServices.SafeHandle.Finalize()
000000bb26d9fbf0 00007ffd68a02ca6 [DebuggerU2MCatchHandlerFrame: 000000bb26d9fbf0]

dotnet/corefx#3: WinHttpHandler.HandleAsyncException ()

Exception object: 000000f740eb92f8
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
System.Net.Http.WinHttpHandler.HandleAsyncException(System.Net.Http.WinHttpRequestState, System.Exception)
System_Net_Http!System.Net.Http.WinHttpHandler+<StartRequest>d__105.MoveNext()+
System_Private_CoreLib!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()System_Private_CoreLib!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System_Private_CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch()

dotnet/corefx#4

Thread Id: 1273 OS Id: 5be0 Locks: 0
Thread is Alive
Last Exception: (System.ExecutionEngineException) (null)

0000000cbaf6deb8 0000000000000000 InlinedCallFrame
0000000cbaf6deb8 0000000000000000 InlinedCallFrame
0000000cbaf6de90 00007ffbd2ac41b3 DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr)
0000000cbaf6df40 00007ffc31906768 Interop+WinHttp+SafeWinHttpHandle.ReleaseHandle()
0000000cbaf6e160 0000000000000000 GCFrame
0000000cbaf6e198 0000000000000000 GCFrame
0000000cbaf6e2e8 0000000000000000 HelperMethodFrame_1OBJ
0000000cbaf6e450 00007ffc31921767 System.Net.Http.WinHttpResponseStream.Dispose(Boolean)
0000000cbaf6e490 00007ffc2dcd009a System.IO.Stream.Close()
0000000cbaf6e4c0 00007ffc31919310 System.Net.Http.NoWriteNoSeekStreamContent+<>c.<SerializeToStreamAsync>b__4_0(System.Threading.Tasks.Task, System.Object)
0000000cbaf6e500 00007ffc2dc3b3be System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0000000cbaf6e570 00007ffc2dcc9d46 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
0000000cbaf6e610 00007ffc2de2c466 System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
0000000cbaf6e660 00007ffc2dcf7213 System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
0000000cbaf6e6c0 00007ffc2dd33a5f System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task, Boolean)
0000000cbaf6e710 00007ffc2dccaf1f System.Threading.Tasks.Task.RunContinuations(System.Object)
0000000cbaf6e800 00007ffc2dd35085 System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].TrySetResult(System.Threading.Tasks.VoidTaskResult)
0000000cbaf6e840 00007ffc2dd114f9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetExistingTaskResult(System.Threading.Tasks.VoidTaskResult)
0000000cbaf6e880 00007ffc2dd1148c System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib]].SetResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>)
0000000cbaf6e8b0 00007ffc31921fc6 System.Net.Http.WinHttpResponseStream+<CopyToAsyncCore>d__18.MoveNext()
0000000cbaf6e970 00007ffc2dc3b3be System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0000000cbaf6e9e0 00007ffc2dc3b3be System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0000000cbaf6ea50 00007ffc2dcc9d46 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
0000000cbaf6eaf0 00007ffc2dd07069 System.Threading.ThreadPoolWorkQueue.Dispatch()

Some Notes on the usage of HttpClient:

• Shared instance of HttpClientHandler as static readonly HttpMessageHandler
• Most often invoked methods:
  • public object Invoke(string method, Type returnType = null, object parameters = null)
  • public async Task<object> InvokeAsync(string method, Type returnType = null, object parameters = null, CancellationToken cancellationToken = default (CancellationToken))
• The core functionality is implemented by async Task<object> GetResponseAsync
• Instantiates new instance of HttpClient on every call as new HttpClient(webServerInvoker.Handler, false) where webServerInvoker.Handler is the shared static instance.
• Awaits for PostAsync result, a timed CancellationToken is provided .
• Always disposes HttpClient instance at the end of the call.
• Note that synchronous public object Invoke enforces its own timeout while waiting for InvokeAsync Task result. It cancels the CancellationTokenSource if the timeout is hit, in turn cancelling any pending HttpClient operation and immediately disposing of HttpClient instance.
• The timeout is provided by the caller. It ranges from 30 seconds to 5 minutes.

[EDIT] Formatting changes by @karelz

[baal2000]

Sharing some interesting errors stacks in addition to already posted. The process crashed while the system was under .Net Core 2.0.0 in a state of high CPU load and lock contention .

• Stack A:

Exception: System.InvalidCastException EXF91F3FC3
Message: Unable to cast object of type 'System.Object[]' to type 'System.Net.Http.WinHttpRequestState'.
 at System.Net.Http.WinHttpRequestState.FromIntPtr(IntPtr gcHandle)
 at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(...)
 at Interop.WinHttp.WinHttpCloseHandle(IntPtr handle)
 at Interop.WinHttp.SafeWinHttpHandle.ReleaseHandle()
 at System.Runtime.InteropServices.SafeHandle.InternalDispose()
 at System.Net.Http.WinHttpHandler.<>c.<StartRequest>b__105_0(Object s)
 at System.Threading.ExecutionContext.Run(...)
 at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)

• Stack B:

Exception: System.NullReferenceException EX1100035F
Message: Object reference not set to an instance of an object.
 at System.Net.Http.WinHttpRequestCallback.RequestCallback(...)
 at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(...)
 at Interop.WinHttp.WinHttpCloseHandle(IntPtr handle)
 at Interop.WinHttp.SafeWinHttpHandle.ReleaseHandle()
 at System.Runtime.InteropServices.SafeHandle.InternalDispose()
 at System.Net.Http.WinHttpHandler.<>c.<StartRequest>b__105_0(Object s)
 at System.Threading.ExecutionContext.Run(...)
 at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)

Discussion

• Stack A:
  The framework pins WinHttpRequestState and retrieves the GCHandle to the instance as _operationHandle = GCHandle.Alloc(this). Then passes GCHandle as GCHandle.ToIntPtr(_operationHandle) through to winhttp API.
  The API callback WinHttpRequestCallback.WinHttpCallback receives IntPtr value of GCHandle, dereferences it to the .Net object instance of class System.Object[], not WinHttpRequestState as expected.
  This can happen if _operationHandle is unpinned early in the API call lifespan. If GC Gen 2 is triggered during that time it collects the instance and moves other heap objects. An unrelated instance can be now found where _operationHandle of WinHttpRequestHandle used to be.

• Stack B:
  Similar scenario. The handle is pointing to an unallocated heap location.

Thanks for looking into that.

[davidsh]

Instantiates new instance of HttpClient on every call as new HttpClient(webServerInvoker.Handler, false) where webServerInvoker.Handler is the shared static instance.

What is the purpose of this pattern? I.e. why have a shared static HttpClientHandler, but yet create a new HttpClient for every request? Why not just create a single HttpClient w/ a handler (HttpClientHandler) that can be shared static?

We are reporting daily crashes after moving our application from .Net 4.6 to .Net Core 2.0.

And was this same pattern being used when the application used .NET Framework 4.x?

[baal2000]

@davidsh

What is the purpose of this pattern

Both approaches (single HttpClient v.s. multiple HttpClients, single HttpMessageHandler) are argued at https://github.com/dotnet/corefx/issues/5811.

Conclusion: HttpClient constrictor supports such pattern, so why not? Is there any reason to not use single HttpClientHandler - multiple HttpClients pattern?

As of Dec 2017 there is known issue #18348 affecting singleton HttpClient. May also affect singleton HttpClientHandler, not sure about that.

2 days ago we updated the build to use static shared HttpClient on most (not every) active parts of the server application, reduced use of canceling of the requests in progress (infinite timeout, no cancellation tokens). 274 servers received the update. Result: 2 servers failed today on a quiet day within 24 hours after upgrade, in line with the usual rate of crashes. We are going to implement this pattern more consistently to see if the static HttpClient is making the system more reliable.

was this same pattern being used when the application used .NET Framework 4.x

Yes, nothing changed.

Thanks.

[davidsh]

As of Dec 2017 there is known issue #18348 affecting singleton HttpClient. May also affect singleton HttpClientHandler, not sure about that.

This doesn't affect HttpClient, per se. It affects HttpClientHandler.

HttpClient is a class that derives from HttpMessageInvoker. HttpMessageInvoker is a simple class that
has one method, a public SendAsync() method. It is the only class that can call the protected internal SendAsync() virtual method of a handler such as HttpClientHandler.

HttpClient itself adds some value over HttpMessageInvoker. It includes several convenience methods like PostAsync, GetAsync. It also has built-in ability to buffer the http response body (HttpCompletionOption enum choices). And it has some properties like HttpClient.Timeout that provide an easy way to set an overall timeout on the HTTP request.

But HttpClient itself is not the HTTP stack. HttpClientHandler is considered the protocol stack. And from a design perspective, things like TCP connection pools, etc. are handled at the HttpClientHandler level.

From a performance perspective, the only real thing of value in sharing objects is the handler. Recycling the HttpClient objects but keeping the HttpClientHandler static isn't really supposed to provide any added value than just creating a static HttpClient with an associated handler that is owned by the HttpClient. In fact, it probably just adds overhead in creating new HttpClient objects each time. So, that is why I asked this question because I don't understand the motivation for creating new HttpClient objects per request yet sharing the same static HttpClientHandler.

[baal2000]

@davidsh

the motivation for creating new HttpClient objects per request yet sharing the same static HttpClientHandler.

There are convenience (HttpClient's individual properties) and historical (rooted in .net 4.0 WebRequest -based code) reasons to use a client instance per request. We realize it can be refactored to the singleton pattern.

Putting aside the best practices: could the multiple client pattern contribute to the crashes in releasing of winhttp handles? If it could: we would accelerate the refactoring efforts. If not: what else shoudl we try as workaround?

Thank you!

[davidsh]

Putting aside the best practices: could the multiple client pattern contribute to the crashes in releasing of winhttp handles?

I think it is possible that this particular pattern is related to the crashes. The pattern is not "illegal" per se, but it is not something that is tested extensively in the current test coverage. It is possible that this pattern is exposing a bug. So, as a workaround, it might be useful to see if changing the pattern helps with reducing these crashes.

[davidsh]

Looking at the HttpClient code, you can see that by recycling (disposing) the HttpClient object, you are essentially cancelling any pending requests that were started by the HttpClient:

https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/HttpClient.cs#L582-L596

HttpClient itself tracks all requests that are pending so that it can support the HttpClient.CancelPendingRequests method. But the current pattern being used here with recycling the HttpClient per request, yet keeping a static HttpClientHandler, is probably contributing to this behavior of crashes. Depending on timing, you are cancelling the requests inadvertently.

So, my early assessment of this is that I think that this pattern is the "cause" of these crashes.

[baal2000]

@davidsh
Makes sense. Hope it would not take long to figure it out.

[neilcawse]

@davidsh @baal2000 David we can confirm that in the change made that was deployed over the weekend we stopped creating any static or shared HttpClientHandlers and we did create a static HttpClients and reused them. In the other places though (less frequently used), we use the HttpClient in a using statement, effectively closing the connection. These two and only ways we are using HttpClient and they are still seeing the very rare AV.

[neilcawse]

@davidsh @stephentoub. Can you check this? This looks like a race condition
If the stream is disposed then _requestHandle.Dispose(); is called and simultaneously a cancellation token is passed to ReadAsync and the task cancelled then CancelPendingResponseStreamReadOperation() calls _requestHandle?.Dispose();

WinHttpResponseStream.cs
Http

        protected override void Dispose(bool disposing)
        {
            if (!_disposed)
            {
                _disposed = true;

                if (disposing)
                {
                    if (_requestHandle != null)
                    {
                        _requestHandle.Dispose();
                        _requestHandle = null;
                    }
                }
            }

            base.Dispose(disposing);
        }

        private void CheckDisposed()
        {
            if (_disposed)
            {
                throw new ObjectDisposedException(this.GetType().FullName);
            }
        }
        
        // The only way to abort pending async operations in WinHTTP is to close the request handle.
        // This causes WinHTTP to cancel any pending I/O and accelerating its callbacks on the handle.
        // This causes our related TaskCompletionSource objects to move to a terminal state.
        //
        // We only want to dispose the handle if we are actually waiting for a pending WinHTTP I/O to complete,
        // meaning that we are await'ing for a Task to complete. While we could simply call dispose without
        // a pending operation, it would cause random failures in the other threads when we expect a valid handle.
        private void CancelPendingResponseStreamReadOperation()
        {
            WinHttpTraceHelper.Trace("WinHttpResponseStream.CancelPendingResponseStreamReadOperation");
            lock (_state.Lock)
            {
                if (_state.AsyncReadInProgress)
                {
                    WinHttpTraceHelper.Trace("WinHttpResponseStream.CancelPendingResponseStreamReadOperation: before dispose");
                    _requestHandle?.Dispose(); // null check necessary to handle race condition between stream disposal and cancellation
                    WinHttpTraceHelper.Trace("WinHttpResponseStream.CancelPendingResponseStreamReadOperation: after dispose");
                }
            }
        }    

[baal2000]

@davidsh @stephentoub
A mentioned above, there is a big difference in the client application behavior between v 2.0.0 (.net core 2.0.0) and .Net 4.6.1 + WinHttpHandler NuGet 4.3 (net standard 1.1.0). The former crashes due to race condition in releasing RequestHandle, the later doesn't.

This commit dotnet/corefx@48214d7 is the biggest departure from 1.1.0. It could've introduced (or enhanced) the issue. Would you recommend to test the previous version of the code or a workaround switch to non-winhttp version of HttpClientHandler?

[tdevere]

Moving this discussion to offline email communication...