Heap corruption when Target OS is set

[uler3161]

Description

I have an ASP.NET Core application, but I don't think it's specific to ASP.NET Core. This app interfaces with GStreamer using GStreamer-Sharp that I've built from source (upgraded with upgrade-assistant and built with .NET 8.0). I have all of this working correctly when I run it from Visual Studio 2022. Target framework for the web app is .NET 8.0.

I have a QA environment set up with Jenkins. It uses msbuild to build and deploy the app to IIS (version 10 on Server 2019). I found that the app worked fine until I tried to run GStreamer functionality. It would kill w3wp.exe with a 0xc0000374 code in ntdll.dll. Apparently that means heap corruption.

Moving back to my development machine, after some experimentation changing settings in VS, I tested the following:

Target OS	Target OS Version	Supported OS Version	Configuration	Platform	Result
(None)			Debug	Any CPU	Works
(None)			Release	Any CPU	Works
Windows	7.0	7.0	Debug	Any CPU	Crashes
Windows	7.0	7.0	Release	Any CPU	Crashes
Windows	10.0.17763.0	10.0.17763.0	Debug	Any CPU	Crashes
Windows	10.0.17763.0	10.0.17763.0	Release	Any CPU	Crashes

Since it appears to be related to GStreamer, I tried playing around with Target OS settings when building GStreamer-Sharp, but that didn't seem to matter. Plus, since it works fine when Target OS is (None), I'm not really sure that's where the problem lies.

I'm making an assumption here that whatever msbuild is doing in my Jenkins deploy is related to the Target OS setting, but I don't know for sure. So perhaps there are two causes here resulting in the same problem. Here is basically what my Jenkins setup is doing:

msbuild.exe path\to\solution.sln /nologo /nr:false /p:platform="Any CPU" /p:configuration="release" /t:clean;restore;rebuild /p:TargetFramework=net8.0
msbuild.exe path\to\webapp.csproj /p:WebPublishMethod=MSDeploy /p:Password=$PASSWORD /p:Username=$USERNAME /p:AllowUntrustedCertificate=true /p:DeployOnBuild=true /p:LaunchSiteAfterPublish=false /p:VisualStudioVersion=17.0 /p:LastUsedBuildConfiguration=Release /p:LastUsedPlatform="Any CPU" /p:SiteUrlToLaunchAfterPublish=https://myserver /p:ExcludeApp_Data=false /p:MSDeployServiceURL=myserver /p:DeployIisAppPath=myapp /p:SkipExtraFilesOnServer=true /p:MSDeployPublishMethod=WMSVC /p:EnableMSDeployBackup=true /p:EnableMsDeployAppOffline=true /p:TargetFramework=net8.0

Reproduction Steps

I've attempted to put together a smaller example program, but haven't been able to replicate the problem. It may be possible to provide our entire application if need be. I also have a dump generated with DebugDiag that I can send along if desired.

Expected behavior

No heap corruption regardless of whether I specify Target OS.

Actual behavior

I collected a dump with DebugDiag. The Analysis pointed this out:

***********************
*  EXCEPTION DETAILS  *
***********************

DetailID = 1
	Count:    1
	Exception #:  0X80000003
	Stack:        
		ntdll!RtlReportCriticalFailure+0x56
		ntdll!RtlpHeapHandleError+0x12
		ntdll!RtlpHpHeapHandleError+0x7a
		ntdll!RtlpLogHeapFailure+0x45
		ntdll!RtlpFreeHeapInternal+0x4e0
		ntdll!RtlFreeHeap+0x51
		ucrtbase!_free_base+0x1b
		gstsdp_1_0_0!gst_sdp_message_free+0x1b
		gstwebrtc_1_0_0!gst_webrtc_session_description_free+0x18
		gobject_2_0_0!g_value_unset+0x29
		gstreamer_1_0_0!gst_structure_free+0x60
		gstreamer_1_0_0!gst_promise_expire+0x163
		gstreamer_1_0_0!gst_mini_object_unref+0x307
		gstwebrtc!gst_plugin_webrtc_register+0x9817
		glib_2_0_0!g_list_sort_with_data+0x231
		glib_2_0_0!g_source_destroy+0xdc
		glib_2_0_0!g_main_depth+0x1ce
		glib_2_0_0!g_main_context_is_owner+0x20a
		glib_2_0_0!g_main_loop_run+0x133
		gstwebrtc!gst_plugin_webrtc_register+0x9fed
		glib_2_0_0!g_thread_new+0x1a2
		glib_2_0_0!g_rw_lock_writer_unlock+0x4a7
		ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
		KERNEL32!BaseThreadInitThunk+0x14
		ntdll!RtlUserThreadStart+0x21


DetailID = 2
	Count:    2
	Exception #:  0XC0000374
	Stack:        
		ntdll!RtlReportFatalFailure+0x9
		ntdll!RtlReportCriticalFailure+0x97
		ntdll!RtlpHeapHandleError+0x12
		ntdll!RtlpHpHeapHandleError+0x7a
		ntdll!RtlpLogHeapFailure+0x45
		ntdll!RtlpFreeHeapInternal+0x4e0
		ntdll!RtlFreeHeap+0x51
		ucrtbase!_free_base+0x1b
		gstsdp_1_0_0!gst_sdp_message_free+0x1b
		gstwebrtc_1_0_0!gst_webrtc_session_description_free+0x18
		gobject_2_0_0!g_value_unset+0x29
		gstreamer_1_0_0!gst_structure_free+0x60
		gstreamer_1_0_0!gst_promise_expire+0x163
		gstreamer_1_0_0!gst_mini_object_unref+0x307
		gstwebrtc!gst_plugin_webrtc_register+0x9817
		glib_2_0_0!g_list_sort_with_data+0x231
		glib_2_0_0!g_source_destroy+0xdc
		glib_2_0_0!g_main_depth+0x1ce
		glib_2_0_0!g_main_context_is_owner+0x20a
		glib_2_0_0!g_main_loop_run+0x133
		gstwebrtc!gst_plugin_webrtc_register+0x9fed
		glib_2_0_0!g_thread_new+0x1a2
		glib_2_0_0!g_rw_lock_writer_unlock+0x4a7
		ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
		KERNEL32!BaseThreadInitThunk+0x14
		ntdll!RtlUserThreadStart+0x21  

Regression?

The app did work previously. I believe with .NET 6.0. If need be, I can try to build an older copy of the code and re-test, but it will take some effort.

Known Workarounds

No response

Configuration

.NET 8.0.
Development machine is Windows 10 x64, Version 22H2, OS build 19045.4170
IIS machine is Server 2019 Standard, Version 1809, OS build 17763.5576
Jenkins machine is Server 2019 Standard, Version 1809, OS build 17763.2090

Other information

I'm curious if #100426 is related.

[jkotas]

I collected a dump with DebugDiag.

These stacktraces suggest that it is a buffer-overrun, double-free or use-after-free bug in gstreamer or the other unmanaged libraries that you depend on in your project. These crashes are not easy to debug.

You can try to debug it yourself (there are number of tutorials for debugging crashes like this) or try to get help in the gstreamer project.

since it works fine when Target OS is (None), I'm not really sure that's where the problem lies.

It is not unusual for bugs like this to come and go due to unrelated changes.

I'm curious if #100426 is related.

It is not related.

[VSadov]

I'm curious if #100426 is related.

It is not related.

#100426 is a relatively rare scenario on NativeAOT. This does not look NativeAOT-specific.

[uler3161]

These stacktraces suggest that it is a buffer-overrun, double-free or use-after-free bug in gstreamer or the other unmanaged libraries that you depend on in your project. These crashes are not easy to debug.

You can try to debug it yourself (there are number of tutorials for debugging crashes like this) or try to get help in the gstreamer project.

Unfortunately my experience with getting help from gstreamer devs is not great when it comes to anything .NET related.

since it works fine when Target OS is (None), I'm not really sure that's where the problem lies.

It is not unusual for bugs like this to come and go due to unrelated changes.

I'm still trying to wrap my head around this. If I were to guess based off the stacktrace, I imagine this is double-free situation. I'm trying to think what the Target OS setting would change. I mean, it seems like the gstreamer code shouldn't be changing. If they are calling free twice, then the same scenario should be happening. Or would it maybe be that Target OS set to (None) somehow doesn't report multiple free calls as a problem?

[jkotas]

The memory heap corruption crashes are often non-deterministic. The double-free can be happening every time, but it does not mean that the program is going to crash every time.

For example, these crashes can come and go depending on the length of path that the executable is run from (ie the executable is exactly same, the only thing changing is the path that it is run from). The longer vs. shorter path changes the exact layout of the address space and that may influence whether the double-free ends up crashing or not.

[uler3161]

The memory heap corruption crashes are often non-deterministic. The double-free can be happening every time, but it does not mean that the program is going to crash every time.

For example, these crashes can come and go depending on the length of path that the executable is run from (ie the executable is exactly same, the only thing changing is the path that it is run from). The longer vs. shorter path changes the exact layout of the address space and that may influence whether the double-free ends up crashing or not.

Ah, I see. That actually helps explain something that was bugging me while trying to test the different combinations of Target OS. I had one configuration that appeared to work, but then I ran it again and it crashed. I assumed I hadn't properly rebuilt it with the Target OS set but I think I did and it just happened to work.

By that logic, even though I've never had an issue when Target OS is set to (None), it's just luck that I haven't seen the issue with that setting?

[jkotas]

By that logic, even though I've never had an issue when Target OS is set to (None), it's just luck that I haven't seen the issue with that setting?

Yes, it is possible.

[uler3161]

By that logic, even though I've never had an issue when Target OS is set to (None), it's just luck that I haven't seen the issue with that setting?

Yes, it is possible.

Thanks for your help. It actually led me to find a problem in my code that fixed this. For reference, here's what happened:

To begin with the back story... Lots of objects in GStreamer-Sharp are IDisposable. If they aren't properly cleaned up, there are massive memory leaks. There are very few examples of how to use this API out there and the ones (official ones I might add) don't seem to ever call Dispose. Based on my experience discussing such things with the GStreamer devs, they expect finalizers to clean everything up after there are no more references. However, I was able to show that in some fairly common scenarios, a third-party library they use is caching objects and, therefore, finalizers never run since the objects can't be marked for GC. The response I seem to always get is that they think the .NET garbage collector isn't very smart.

Where that left me was randomly choosing which IDisposables to call Dispose on as well as figuring out when I needed to make the call. There were two Dispose calls I removed and that fixed the problem. I haven't yet verified if this re-introduces some memory leaks, so I'm not 100% sure things are fixed, but at least it's not crashing with heap corruption. Regardless, I'm closing the issue since it isn't a .NET issue.