Cannot access a disposed object. Object name: 'SslStream'.

[samneefs]

I have an issue with SendAsync in .NET Core 2.2.
When doing a lot of requests, I encounter the exception on some of them.
I have gotten the error on different requests, for example on a small GET request. However I do have to mention that asynchronously I'm also doing other requests, for example POST (with content smaller than 4MB).

This issue is similar to https://github.com/dotnet/corefx/issues/34033 , but not the same. Mine doesn't mention the handshake in the stacktrace and the other one should be fixed since Core 2.1 (and I use 2.2).

The operation was canceled.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at [details of own code omitted]

Inner exception:
The read operation failed, see inner exception.
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)

Inner Exception:
Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)

[scalablecory]

Is it possible your requests are timing out? Does the same issue happen on .NET Core 3.0?

[Plasma]

Are you accidentally returning from an async method without awaiting within a using block?

[samneefs]

@scalablecory
Some sort of timeout seems likely:
when I do another request, that is one single long running request, to the same server I get a similar error after 10min, however, this error is not the same (and strangely it stays 10min after changing the timeout to something bigger).
However increasing the timeout on the client (to 20 min), had no effect: still got errors before the timeout.
On the server the executionTimeout had been increased tot 1800s (30min) and it also triggered before this.
About the original calls:
When doing all calls synchronously after each other (instead of async, multiple together), then I don't encounter the problem.
Maybe I created to much load on the server when I do it async and this breaks the connection?
I did await Task.WhenAll(tasks); maybe there is a timeout on how long I can await a task?

@Plasma
No, I receive a task for each request and await them all with await Task.WhenAll(tasks);

[samneefs]

@scalablecory
With the long running request, the exception is a bit different. Here are the details:

An error occurred while sending the request.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at SBB.Exact.Rest.Proxy.Handwritten.BaseExactRestApiProxy.GetAsync[T](String url, String applicationId, String division)
   at SBB.Exact.Rest.Proxy.Handwritten.BaseExactRestApiProxy.GetAllAsync[T](String applicationId, String division, String[] fields, String where, Boolean withPagination, String skiptoken, Int32 numberOfPages)
   at SBB.Exact.Rest.Proxy.Handwritten.ExactRestApiProxy.GetBulkTransactionLinesAsync(String applicationId, String division, String[] fields, String where)
   at Winbooks2Exact.Core.Tests.IntegrationTests.ExactRestIntegrationTest.GetAllTransactionLines(String dossierNr, String divisionId) in C:\Workspaces\SelectProjects\Winbooks3\Winbooks2Exact\Dev\SBB.Winbooks2Exact\Winbooks2Exact.Core.Tests\IntegrationTests\ExactRestIntegrationTest.cs:line 29
      
InnerException	
   Unable to read data from the transport connection: De externe host heeft een verbinding verbroken.
      at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
     
InnerException	
{System.Net.Sockets.SocketException (10054): De externe host heeft een verbinding verbroken}	

[scalablecory]

Okay, can you reduce your code into a small repro? This will help diagnose the problem.

[davidsh]

@samneefs
I think if the only exception were this:

The operation was canceled.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at [details of own code omitted]

then this would be perceived as "normal" behavior. By "normal", I mean that the HTTP request was cancelled probably due to a timeout-related cancellation token expiring. The cancellation token could either be one you pass into the HttpClient APIs. Or the internal one used when HttpClient.Timeout is used. By default HttpClient.Timeout has a non-infinite timeout.

I think the "bug" here is that while cancelling the HTTP request, there are operations that get cancelled internally (such as on SslStream). And cancelling those I/O operation results in a bug where we are trying to reference the, now disposed, SslStream. That is probably why those other internal exceptions are occurring.

It would probably make sense for the HTTP stack to potentially ignore some of those exceptions since they are "expected" to occur while shutting down the connection. However, I think the SslStream ObjectDisposedException is something where we can fix that so that we don't try to access the SslStream after it has been disposed.

[wfurt]

I got same exception in the local build

      System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
      ---- System.TimeoutException : The operation was canceled.
      -------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
      ------------ System.Net.Http.HttpRequestException : Error while copying content to a stream.
      ---------------- System.ObjectDisposedException : Cannot access a disposed object.
      Object name: 'SslStream'.
      Stack Trace:
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(564,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/PostScenarioTest.cs(242,0): at System.Net.Http.Functional.Tests.PostScenarioTest.PostHelper(RemoteServer remoteServer, String requestBody, HttpContent requestContent, Boolean useContentLengthUpload, Boolean useChunkedEncodingUpload)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/PostScenarioTest.cs(157,0): at System.Net.Http.Functional.Tests.PostScenarioTest.PostLargeContentUsingContentLengthSemantics_Success(RemoteServer remoteServer, Int32 contentLength)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----

        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(695,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1008,0): at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(734,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(33,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTestBase.cs(104,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTestBase.VersionCheckerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(542,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(378,0): at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(759,0): at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(444,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(832,0): at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|140_0(ExceptionDispatchInfo e)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(822,0): at System.Net.Security.SslStream.ThrowIfExceptional()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs(797,0): at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1212,0): at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1196,0): at System.Net.Http.HttpConnection.FlushAsync()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(982,0): at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(374,0): at System.Net.Http.HttpContent.CopyToAsyncCore(ValueTask copyTask)
  Finished:    System.Net.Http.Functional.Tests
Canceling due to test failure...
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 1831, Errors: 0, Failed: 2, Skipped: 54, Time: 204.718s

I was running tests under strace and everything is very slow. I did get several OperationCanceled exceptions before (and I fiddle with timeouts) but this is the first time when I get ObjectDisposedException.

[wfurt]

I can also get running current 5.0 master.

     Condition(s) not met: "IsWindows10Version1607OrGreater"
      System.Threading.Tasks.TaskCanceledException : The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
      ---- System.TimeoutException : The operation was canceled.
      -------- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
      ------------ System.ObjectDisposedException : Cannot access a disposed object.
      Object name: 'System.Net.Sockets.NetworkStream'.
      Stack Trace:
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(564,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        /mnt/github/wfurt-runtime2/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.AutoRedirect.cs(359,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_AutoRedirect.GetAsync_MaxAutomaticRedirectionsNServerHops_ThrowsIfTooMany(Int32 maxHops, Int32 hops)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----

        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(695,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1008,0): at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(734,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(81,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(542,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
        ----- Inner Stack Trace -----
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(875,0): at System.Net.Sockets.NetworkStream.<ThrowIfDisposed>g__ThrowObjectDisposedException|63_0()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(872,0): at System.Net.Sockets.NetworkStream.ThrowIfDisposed()
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs(801,0): at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1212,0): at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(1119,0): at System.Net.Http.HttpConnection.WriteBytesSlowAsync(Byte[] bytes)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(263,0): at System.Net.Http.HttpConnection.WriteHostHeaderAsync(Uri uri)
        /mnt/github/wfurt-runtime2/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs(402,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
  Finished:    System.Net.Http.Functional.Tests
Canceling due to test failure...
=== TEST EXECUTION SUMMARY ===
   System.Net.Http.Functional.Tests  Total: 1729, Errors: 0, Failed: 3, Skipped: 54, Time: 196.215s

[davidsh]

I would say based on this stack trace, the only part we need to fix is the 'ObjectDisposedException'. We should be able to detect when the SslStream is being closed by upstack code (in this case due to a cancellation) and prevent SslStream from accessing its own objects while closing down.

[karelz]

Triage: We should see if we can detect it and avoid throwing. If not, then it is likely confusing to customers, in which case we may want to mask it / not include as inner exception of TaskCanceled.

Note: We have seen more customers running into this and being confused in the past.