SafeHandle use-after-dispose in FileSystemWatcher on OSX

[stephentoub]

OSX.1013.Amd64.Open-x64:Debug
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38953~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190626.70/workItem/System.IO.FileSystem.Watcher.Tests/wilogs

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Interop.RunLoop.CFRunLoopRun()
   at System.IO.FileSystemWatcher.RunningInstance.StaticWatcherRunLoopManager.WatchForFileSystemEventsThreadStart(Object args) in /_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.OSX.cs:line 247
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 201
Fatal error. Internal CLR error. (0x80131506)
   at Interop+RunLoop.CFRunLoopRun()
   at Interop+RunLoop.CFRunLoopRun()
   at System.IO.FileSystemWatcher+RunningInstance+StaticWatcherRunLoopManager.WatchForFileSystemEventsThreadStart(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

cc: @JeremyKuhne, @carlossanlop

Runfo Tracking Issue: system.io.filesystem.watcher.tests work item

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1082967	runtime	PR 51078	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1082899	runtime	PR 51099	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88	console.log	core dump
1082624	runtime	PR 51090	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1080937	runtime	Rolling	net5.0-Linux-Release-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-bfcd90a-20200127194925	console.log			runclient.py
1072066	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88	console.log	core dump		runclient.py
1072066	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump
1071508	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88	console.log	core dump
1071508	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump
1069555	runtime	Rolling	net6.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log			runclient.py
1069328	runtime	Rolling	net6.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open	console.log			runclient.py
1068131	runtime	PR 50257	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1066426	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88	console.log	core dump
1066426	runtime	PR 50364	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump
1066285	runtime	PR 50507	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log			runclient.py
1065570	runtime	PR 50483	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1065167	runtime	Rolling	net6.0-OSX-Release-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log	core dump		runclient.py
1063683	runtime	PR 48527	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1059736	runtime	PR 50232	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1059736	runtime	PR 50232	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log			runclient.py
1059736	runtime	PR 50232	net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1059736	runtime	PR 50232	net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59	console.log	core dump		runclient.py
1058168	runtime	PR 50280	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1057844	runtime	PR 50275	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1055598	runtime	Rolling	net6.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1050603	runtime	PR 49990	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log
1050314	runtime	PR 48601	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log			runclient.py
1049560	runtime	Rolling	net6.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1047468	runtime	PR 49545	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1047249	runtime	Rolling	net6.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-mono_interpreter_release-Debian.9.Amd64.Open	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-(Centos.8.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open	console.log	core dump		runclient.py
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-SLES.15.Amd64.Open	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-x64-Mono_release-(Fedora.30.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7	console.log	core dump
1046026	runtime	PR 49740	net6.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673	console.log	core dump
1046026	runtime	PR 49740	net6.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open	console.log	core dump
1046026	runtime	PR 49740	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log	core dump
1045010	runtime	PR 48369	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log			runclient.py
1044737	runtime	PR 49373	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log	core dump		runclient.py
1044726	runtime	PR 49532	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1043361	runtime	PR 49747	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1042998	runtime	PR 49741	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1041823	runtime	PR 49251	net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open	console.log			runclient.py
1039858	runtime	PR 43706	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.312.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88	console.log	core dump
1039858	runtime	PR 43706	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump
1039185	runtime	Rolling	net6.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1038808	runtime	PR 49613	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1037652	runtime	PR 49529	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1037058	runtime	PR 41097	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log	core dump		runclient.py

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
3	4	33

[stephentoub]

cc: @MaximLipnin
Maybe related to dotnet/corefx#33865

[danmoseley]

@stephentoub do you know what could cause a NRE in a native call into macOS? Is the interop layer translating some kind of errno returned from the OS here?

Then I see we get a COR_E_EXECUTIONENGINE while trying to deal with the exception.

[stephentoub]

Not sure; @jkoritzinsky, @AaronRobinsonMSFT, ideas?

[AaronRobinsonMSFT]

@danmosemsft and @stephentoub The COR_E_EXECUTIONENGINE can be caused by an invalid P/Invoke stub or invalid translation as well. I did recently fix a few issues in macOS regarding race conditions in shutdown - dotnet/coreclr#24847. How often is this happening? dotnet/corefx#33865 does look promising too. Can someone point to the P/Invoke that is causing this issue?

[stephentoub]

@AaronRobinsonMSFT, I only saw it that once. This is the P/Invoke:
https://github.com/dotnet/corefx/blob/2aa2518a47c541148e2dc3f9d040942e44395acd/src/Common/src/Interop/OSX/Interop.RunLoop.cs#L33-L35
where CoreFoundationLibrary is:
https://github.com/dotnet/corefx/blob/2aa2518a47c541148e2dc3f9d040942e44395acd/src/Common/src/Interop/OSX/Interop.Libraries.cs#L9

[danmoseley]

For future reference, it seems console output cannot directly be queried in Kusto. The best we can easily do is look for 134

WorkItems 
| where Started > now(-10d)  
| where FriendlyName == "System.IO.FileSystem.Watcher.Tests"
| where ExitCode == 134
| join kind= inner (
   Jobs  | where Started > now(-10d) | project  QueueName , JobId, Build, Type, Source 
) on JobId
| summarize count() by ExitCode, QueueName
| order by count_ desc

ExitCode QueueName count_
134 Ubuntu.1604.Arm32.Open 28
134 Ubuntu.1604.Arm64.Docker.Open 25
134 Ubuntu.1804.Amd64.Open 6
134 OSX.1013.Amd64.Open 1

compared to ~5000 with 0 exit code.

[AaronRobinsonMSFT]

@danmosemsft Do we know what version of coreclr is being used to run the tests themselves? Is it possible the coreclr being used in this instance is older than the date at which dotnet/coreclr#24847 was checked in?

[stephentoub]

We'd have to dig to see which exact version, but this was 5 days ago, and corefx ingested a new coreclr 5 days ago, 6 days ago, and 7 days ago, so it definitely has that change from end of May.

[danmoseley]

@mjanecke can you tell from the output at https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38953~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190626.70/workItem/System.IO.FileSystem.Watcher.Tests/wilogs whether a dump got uploaded? I don't see the usual dumpling song and dance.

[missymessa]

@danmosemsft, for that test, it looks like the failure happened on a Mac, which I don't think @epananth has finished implementing for dump collection yet.

[danmoseley]

@mjanecke good point, and last I heard we did not have a way to create a dump on Mac (that could be debugged with managed debuggers). @mikem8361 any update on that? Is our only option to catch this under an attached debugger?

(It's quite possible this isn't Mac specific actually but I'm interested anyway)

[mikem8361]

The best way would be to catch this under a managed debugger. There is no “createdump” facility on macOS. You can setup your macOS to generate system core dumps (ulimit -c unlimited, etc.) but there a limitation in lldb that makes SOS on coredumps on macOS difficult. See https://github.com/dotnet/diagnostics/blob/master/documentation/debugging-coredump.md#launch-lldb-under-macos. mikem

[danmoseley]

Since this is clearly mac specific we can confidently say it has only happened 1 time in 10 days.

[danmoseley]

No more hits in the last week. Query:

WorkItems 
| where Started > now(-10d)  
| where FriendlyName == "System.IO.FileSystem.Watcher.Tests"
| where ExitCode == 134
| join kind= inner (
   Jobs  | where Started > now(-10d) | project  QueueName , JobId, Build, Type, Source 
) on JobId
| summarize count() by ExitCode, QueueName
| order by count_ desc

	ExitCode	QueueName	count_
	134	Ubuntu.1604.Arm32.Open	3

[carlossanlop]

Seems to be infrastructure related. Moving to 5.0.