.NET Runtime Exception: ObjectDisposedException

[sepehr1014]

Describe the bug

After upgrading to ASP.NET Core 2.2, we frequently face this error and our website gets restarted. This happens once an hour or so. We had to target 2.1 again to fix this issue.
Here are the errors from event viewer (in the order they have occurred):

Application: dotnet.exe
CoreCLR Version: 4.6.27110.4
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.IO.FileStream.FileStreamCompletionSource.CompleteCallback(UInt64 packedResult)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Faulting application name: dotnet.exe, version: 2.2.27110.6, time stamp: 0x5be769ed
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2457, time stamp: 0x5b7e2adb
Exception code: 0xe0434352
Fault offset: 0x0000000000033c58
Faulting process id: 0x34374
Faulting application start time: 0x01d48cab8dc6d688
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Faulting package full name: 
Faulting package-relative application ID: 

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_ade641d7

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 4
Hashed bucket: 

Fault bucket 1396378374383491433, type 5
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_69a648dc

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 0
Hashed bucket: 0fdfa957dcca13d25360eebbb12a5969

To Reproduce

Steps to reproduce the behavior:
Use SDK v2.2.100 with IIS on Windows Server 2016

Additional context

Add any other context about the problem here.
Include the output of dotnet --info

Host (useful for support):
  Version: 2.2.0
  Commit:  1249f08fed

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[Tratcher]

Do you have any idea what application code or event triggered this?

This may need to be moved to https://github.com/dotnet/corefx/issues.

[sepehr1014]

@Tratcher Unfortunately not. I was hoping the stack trace would shed some light for you guys. Is it related to files and streams? We're using static files module but there are few actions that return 'File'. We also make extended use of web sockets.

[RoySalisbury]

I am also getting this. Just start the app and refresh a web page a few times and get the

System.ObjectDisposedException: 'The CancellationTokenSource has been disposed.'

Nothing special about my web app as far as I can tell. I thought it was my separate hosted service, but I took that out and it still has the issue.

[RoySalisbury]

Pretty sure this has nothing to do with it, but I removed

app.UseStaticFiles()

And I no longer have the issue. Could just be taking longer now, -or- it has something to do with the loading of css/js files....

[RoySalisbury]

OK.. Left in the 'UseStaticFiles' call but removed every js/css file being loaded. Still happened. But removed the 'UseStaticFiles' and after 10 min of constant refresh, no error. Usually it will happen within 4 or 5 page refreshes.

So, now, what is the call to 'UseStaticFiles' doing that either causes it, or makes it happen?

[Tratcher]

StaticFiles passes the HttpContext.RequestAborted CancellationToken to the read and write operations when it's serving a file. When you repeatedly press F5 some operations do not complete and get aborted. That's standard stuff and nothing has changed there in a long time.

What might be new in 2.2: IIS applications are hosted in-process by default (as opposed to the prior out-of-process mode). It may be disposing of the RequestAborted CTS after canceling it. @jkotalik an aborted CTS should not be disposed, it causes races like this.

Can you reproduce this if you edit the csproj and change it from InProcess to OutOfProcess? Or if you run only with Kestrel?

Reguardless, there's a bug in CoreFx FileStreamCompletionSource about throwing exceptions on background threads that we should file.

[jkotalik]

@Tratcher I think @sepehr1014 issue isn't related to inproc as the faulting exe is dotnet.exe.

I looked at how we handle request aborted in InProc and from what I can tell, we don't call dispose on the cts after cancelling it. We can add some debug asserts to confirm.

[RoySalisbury]

I'm actually running with Kestrel. And even waiting between refreshes of the page still causes the issue. So I don't think its the 'F5' aborting things.

[Tratcher]

I did find what looks like a regression in FileStreamCompletionSource (need to confirm which versions), but that doesn't explain why the CTS was disposed.

[Tratcher]

Hmm, the FileStreamCompletionSource happened in 2.1, not 2.2. There may be an additional change in Kestrel's CTS handling in 2.2...

[RoySalisbury]

Looking at the changes linked above, the only thing that jumped out at me was the following:

 // Free up the native resource and cancellation registration
 CancellationToken cancellationToken = _cancellationRegistration.Token; // access before disposing registration
 ReleaseNativeResource();

And that's only because in ReleaseNativeResource() it calls cancellationRegistration.Dispose() and I am not sure what that call does to the internals of the original Token reference. Could it be doing something to the original token?

I think I am just going in circles now. I traced back some discussions that may or my not be relevant..

https://github.com/dotnet/corefx/issues/14903

There was a mention in that thread about how Kestral does something different to handle an issue. Then once I dug into that I found this nugget:

           public void TryDispose()
           {
                // This normally waits until the callback is finished invoked but we don't care
                if (_callbackWrapper.TrySetInvoked())
                {
                    // Bug #1549, .NET 4.0 has a bug where this throws if the CTS
                    _registration.Dispose();
                }
            }

So your guess that Kestrel does something different is valid. They have an entire extension class just for CancellationTokens.

I switched over to IISExpress on my dev box and the issue seems to have gone away (another indication that its Kestrel). My end product is AM32 (Pi) which will be Kestrel, so I will do some testing there to see if its a platform specific thing.

However, MY issues seems that it has diverged from the original issue because he WAS getting it to happen in IIS and not Kestrel. Could be that the UseStaticFiles on Kestrel just made it show up quicker and something else in IIS made it happen (but for the same underlying cause).