[6.0.100-preview.4.21206.10][main branch] LibGit2Sharp System.NullReferenceException at git_diff_find_similar native method

[jiangzeng01]

Application Name: LibGit2Sharp (https://github.com/libgit2/libgit2sharp)
OS: Windows 10 RS5
CPU: X64
.NET Build Number: 6.0.100-preview.4.21206.10
Affected Application: git-istage global tool app (GlobalToolsNETCore21Part01)
https://github.com/terrajobst/git-istage

Verify Scenarios:
1). Windows 10 RS5 X64 + .NET Core SDK build 6.0.100-preview.4.21206.1 - main branch: Fail
2). Windows 10 RS5 X64 + .NET Core SDK build 6.0.100-preview.3.21202.5 - Preview 3 branch: Pass
3). Windows 10 RS5 X64 + .NET Core SDK build 5.0.104 : Pass

DevDiv bug: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1307298

Repro steps to run

1. Create a .net core console app
2. Add <PackageReference Include="LibGit2Sharp" Version="0.26.2" />
3. In Program.cs

var _repository = new Repository(@".git folder path in the project");
var tipTree = _repository.Head.Tip?.Tree;
var comp = _repository.Diff.Compare<TreeChanges>(tipTree, DiffTargets.Index);  //exception here
Console.WriteLine(comp.Count); //changes count

Expected Result:
Should return the number of staged changes

Actual Result:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Object.GetType()
   at LibGit2Sharp.Core.NativeMethods.git_diff_find_similar(git_diff* diff, GitDiffFindOptions options)
   at LibGit2Sharp.Core.Proxy.git_diff_find_similar(DiffHandle diff, GitDiffFindOptions options)
   at LibGit2Sharp.Diff.DetectRenames(DiffHandle diffList, CompareOptions compareOptions)
   at LibGit2Sharp.Diff.BuildDiffList(ObjectId oldTreeId, ObjectId newTreeId, TreeComparisonHandleRetriever comparisonHandleRetriever, DiffModifiers diffOptions, IEnumerable`1 paths, ExplicitPathsOptions explicitPathsOptions, CompareOptions compareOptions)
   at LibGit2Sharp.Diff.Compare[T](Tree oldTree, DiffTargets diffTargets, IEnumerable`1 paths, ExplicitPathsOptions explicitPathsOptions, CompareOptions compareOptions)
   at LibGit2Sharp.Diff.Compare[T](Tree oldTree, DiffTargets diffTargets)
   at ConsoleApp1.Program.Main(String[] args) in C:\Users\v-altunc\source\repos\ConsoleApp12\ConsoleApp1\Program.cs:line 14

@dotnet-actwx-bot @dotnet/compat

[jiangzeng01]

@jkoritzinsky @AaronRobinsonMSFT @elinor-fung This issue happened after the issue #50410 was fixed.
Could you please check about this?

[AaronRobinsonMSFT]

@jiangzeng01 Yes, we did observe this issue and I believe @jkoritzinsky has addressed that in our .NET 5 servicing and also recently put in a fix for .NET 6.

[AaronRobinsonMSFT]

See #50735.

[jiangzeng01]

@AaronRobinsonMSFT Thanks! We will verify this with main branch builds.

[jiangzeng01]

@AaronRobinsonMSFT This case is still failing today's latest SDK of main branch: dotnet-sdk-6.0.100-preview.4.21210.1. Could you please take a look?

[AaronRobinsonMSFT]

@jiangzeng01 Thanks. This will be investigated today.

[jkoritzinsky]

The latest SDK build from dotnet/installer is still using a runtime build from 7 days ago before the fix was merged.

Once dotnet/installer takes an updated copy of dotnet/runtime, then the fix will be in a full sdk build.

[jkoritzinsky]

Specifically, this is blocked on dotnet/installer#10185, which is blocked on dotnet/sdk#16810. The dotnet/installer configuration requires the runtime to match the one the SDK was built with and dotnet/installer#10185 needs #51004, whose uptake into dotnet/sdk is part of dotnet/sdk#16810

[jiangzeng01]

@jkoritzinsky @AaronRobinsonMSFT Thanks for update, we will check after runtime build is updated in dotnet/installer.

[jiangzeng01]

We were testing on Preview 3 builds last week, and now on main branch build.
Similar issue found with following 10 apps in main branch build - dotnet-sdk-6.0.100-preview.4.21210.1:
RMMVCookTool, ILA, Rail, FileRenamerDiff, SCM2020, ffmediaelement, DiscordChatExporter, PresentationTheme Aero, Epub, REvernus

Do you think they are same issue with this ticket? @jkoritzinsky

Error log example 1:

C:\Users\appcompat\Downloads\REvernusPayload\applications\REvernus>dotnet REvernus.dll
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Object.GetType()
   at MS.Internal.AppModel.ITaskbarList3.SetThumbnailClip(IntPtr hwnd, RefRECT prcClip)
   at System.Windows.Window.UpdateTaskbarThumbnailClipping()
   at System.Windows.Window.ApplyTaskbarItemInfo()
   at System.Windows.Window.WindowFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at System.Windows.Interop.HwndSource.PublicHooksFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
   at System.Windows.Threading.Dispatcher.PushFrame(DispatcherFrame frame)
   at System.Windows.Threading.Dispatcher.Run()
   at System.Windows.Application.RunDispatcher(Object ignore)
   at System.Windows.Application.RunInternal(Window window)
   at System.Windows.Application.Run()
   at REvernus.App.Main()

Error log example 2:

C:\Users\appcompat\Downloads\ILAPayload (2)\applications\ILA>dotnet ilaGUI.dll
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Object.GetType()
   at MS.Win32.UnsafeNativeMethods.UpdateLayeredWindow(IntPtr hwnd, IntPtr hdcDst, POINT pptDst, POINT pSizeDst, IntPtr hdcSrc, POINT pptSrc, Int32 crKey, BLENDFUNCTION& pBlend, Int32 dwFlags)
   at System.Windows.SplashScreen.Fadeout_Tick(Object unused, EventArgs args)
   at System.Windows.Threading.DispatcherTimer.FireTick(Object unused)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(Object state)
   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Windows.Threading.DispatcherOperation.Invoke()
   at System.Windows.Threading.Dispatcher.ProcessQueue()
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
   at System.Windows.Threading.Dispatcher.PushFrame(DispatcherFrame frame)
   at System.Windows.Threading.Dispatcher.Run()
   at System.Windows.Application.RunDispatcher(Object ignore)
   at System.Windows.Application.RunInternal(Window window)
   at System.Windows.Application.Run()
   at ilaGUI.App.Main()

[jkoritzinsky]

These look like the same issue to me.

[AaronRobinsonMSFT]

Similar issue found with following 10 apps in main branch build - dotnet-sdk-6.0.100-preview.4.21210.1:
RMMVCookTool, ILA, Rail, FileRenamerDiff, SCM2020, ffmediaelement, DiscordChatExporter, PresentationTheme Aero, Epub, REvernus

@jiangzeng01 Thank you for following up with these details. Understanding this sort of breadth of impact is key to how we think about making Interop code changes.

@jkoritzinsky and @elinor-fung I think we should set aside some time to fill out our compat matrix. The above impact makes me think we are getting to point where we just don't touch the built-in system unless it is security related.

[jkoritzinsky]

I'm definitely feeling the same way.

[jiangzeng01]

Verified with dotnet-sdk-6.0.100-preview.4.21215.11, affected cases passed. Close this ticket as fixed.