Crash when using WebListener and Negotiate auth scheme

[jchannon]

Scenario :

Server is running with options.ListenerSettings.Authentication.Schemes = AuthenticationSchemes.Negotiate;

Client IE settings to automatically logon sites in the intranet zone. URL of site is not in the the sites list.

Client makes request to server and it crashes the server. Put the URL in the sites and auth takes place and all is well.

We ran DebugDiag to try and get a crash dump and below is the stack trace inside it (note this is reproducible on a customer installation currently 😢 ) :

An unhandled exception of type 'System.EntryPointNotFoundException' occurred in Microsoft.Net.Http.Server.dll

Additional information: Unable to find an entry point named 'CancelIoEx' in DLL 'api-ms-win-core-io-l1-1-1.dll'.


    Microsoft.Net.Http.Server.dll!Microsoft.Net.Http.Server.RequestContext.ForceCancelRequest()    Unknown
    Microsoft.Net.Http.Server.dll!Microsoft.Net.Http.Server.RequestContext.Abort()    Unknown
    Microsoft.Net.Http.Server.dll!Microsoft.Net.Http.Server.ResponseStreamAsyncResult.Fail(System.Exception ex)    Unknown
    Microsoft.Net.Http.Server.dll!Microsoft.Net.Http.Server.ResponseStreamAsyncResult.IOCompleted(Microsoft.Net.Http.Server.ResponseStreamAsyncResult asyncResult, uint errorCode, uint numBytes)    Unknown
    System.Private.CoreLib.ni.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP)    Unknown

[Tratcher]

OS? Windows 7 / 2008 R2?
Framework? .NET or .NET Core?
WebListener version?

api-ms-win-core-io-l1-1-1.dll is a dependency not present on older Windows versions but can be bundled or installed. I'll work on a repro and a workaround.

[moozzyk]

(or in case of standalone applications the application was published for a newer version of Windows that it is running on e.g. an app published for win8-x64 being run on win7-x64)

[jchannon]

It is a self contained app published with win7-x64

It is .net core "Microsoft.AspNetCore.Server.WebListener": "1.1.0",

Running on Windows 2008 R2 SP1

[jchannon]

Just a bit more info from the customer.

Their Local Intranet settings does in fact have a *.enterprise.com wildcard in Site List but when they visit https://server.enterprise.com we see the above stack trace. On a machine where they can edit these settings if they directly put https://server.enterprise.com in the Site List all is well.

[Tratcher]

That call stack looks like a secondary exception (which we should fix), but it doesn't tell us what the primary exception was. Can they reproduce it reliably enough to capture the first chance exception with the debugger?

[Tratcher]

Since it's a self-contained app, do you have 'api-ms-win-core-io-l1-1-1.dll' in your app directory?

[jchannon]

The dll is there. I can send the dump file if you like?

…

On Fri, 17 Mar 2017 at 19:35, Chris R ***@***.***> wrote: Since it's a self-contained app, do you have 'api-ms-win-core-io-l1-1-1.dll' in your app directory? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGaps-SGTZoOt6qjXOUXzLMaguodJIfks5rmuCZgaJpZM4Md5OB> .

[jchannon]

The dump file is here - https://drive.google.com/file/d/0B4zNlOLPdHfeYVdMT2lrT1FVOE0/view?usp=sharing

[jchannon]

Any update on this, customer = large international bank

[Tratcher]

@jchannon urgent customer issues should be routed to Microsoft Support (https://support.microsoft.com/). The dev team does not have the bandwidth for all investigations.

[Tratcher]

I found the primary exception in the dump:

System.EntryPointNotFoundException: Unable to find an entry point named 'CancelIoEx' in DLL 'api-ms-win-core-io-l1-1-1.dll'.
   at Microsoft.Net.Http.Server.UnsafeNclNativeMethods.CancelIoEx(SafeHandle handle, SafeNativeOverlapped overlapped)
   at Microsoft.Net.Http.Server.RequestContext.ForceCancelRequest()
   at Microsoft.Net.Http.Server.RequestContext.Abort()
   at Microsoft.Net.Http.Server.ResponseStreamAsyncResult.FailSilently()
   at Microsoft.Net.Http.Server.ResponseStreamAsyncResult.IOCompleted(ResponseStreamAsyncResult asyncResult, UInt32 errorCode, UInt32 numBytes)}	System.Exception {System.EntryPointNotFoundException}

It's actually the same error triggered slightly earlier:

HttpSysServer/src/Microsoft.AspNetCore.Server.HttpSys/RequestProcessing/ResponseStreamAsyncResult.cs

Line 247 in 8d61026

asyncResult.FailSilently();

vs

HttpSysServer/src/Microsoft.AspNetCore.Server.HttpSys/RequestProcessing/ResponseStreamAsyncResult.cs

Line 270 in 8d61026

asyncResult.Fail(e);

FailSilently would be called when the client disconnected before the response was completed. It's not clear why this would be related to Negotiate, unless they were writing their own 401 error page?

However, it still comes down to not being able to load that dll. I have not been able to reproduce this issue with the 1.1.1 runtime, dev bits, and Win7x64. If you enable COREHOST_TRACE=1 is the specified dll in the trace? You should see something like Probed deps json and matched [C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.1.1\api-ms-win-core-io-l1-1-1.dll]

[jchannon]

Thanks will get this setup on the customer's install.

Our application does have middleware that does return 401 in a couple of locations eg/XSRF checks, user validation

[jchannon]

Here's a log with the env var set. Thanks

https://drive.google.com/file/d/0B3uxSfah57yBamxoTWhmMzR6cDA/view?usp=sharing

[jchannon]

Any update? Thanks 🙏

[Tratcher]

The log says the file was loaded but that it couldn't find the method within the file. I was not able to reproduce the issue. I don't think we'll be able to make any more progress here without directly debugging the customers app (or related repro app). Did you contact support? They're better set up for direct customer contact.

[jchannon]

Thanks.

Just FYI, this is what I've got back from them

[mikeh688]

I didnt yet contact MS support because in the past I have found it impenetrable. Sorry for the painfully basic question but what category will .Net Core fall into? Our customer is a very large Microsoft customer and ideally we'd like to be able to ask them to have one of their MS contacts step in and help us negotiate our way thru what needs doing.

[Tratcher]

I'm not sure how support is structured, but organizationally .NET Core lives under Visual Studio, which is under Cloud and Enterprise.

[muratg]

@livarcocc Should we move this bug to SDK or CLI under dotnet?

[livarcocc]

@muratg At first sight, this seems more like a corefx issue.

@Petermarcu can you help take a look at this?

[Petermarcu]

@gkhanna79 to help out.

[Petermarcu]

I agree though, this may be better served going through support.

[gkhanna79]

@Tratcher is correct that this binary was loaded - this is what was in the log:

[24/03/2017 18:52:41]
  Thread created. New thread - System ID: 4744
  C:\Program Files\Acano Manager\Portal\api-ms-win-core-io-l1-1-1.dll loaded at 0xf0d50000

The exception above is self-explanatory - the binary in question did not export the API. Can you open a VS Developer Command prompt and do the following against the above binary:

dumpbin /exports C:\Program Files\Acano Manager\Portal\api-ms-win-core-io-l1-1-1.dll

This will list the APIs exported by the above DLL and should confirm that CancelIOEx is not present in it. If confirmed, we will need to understand why Microsoft.Net.Http.Server.UnsafeNclNativeMethods.CancelIoEx expects that API to be present in the above binary.

@jchannon Can you help with the above data collection? Also, what version of .NET Core is this on?

[Tratcher]

API Set docs: https://msdn.microsoft.com/en-us/library/windows/desktop/mt186421(v=vs.85).aspx

We have not been able to reproduce this issue on other machines.

[jchannon]

@gkhanna79 Below is the output you asked for.

c:\AM 2.3 (58fad7)-windows\portal>dumpbin /exports api-ms-win-core-io-l1-1-1.dll
Microsoft (R) COFF/PE Dumper Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file api-ms-win-core-io-l1-1-1.dll

File Type: DLL

  Section contains the following exports for api-ms-win-core-io-l1-1-1.dll

    00000000 characteristics
    5485138C time date stamp Mon Dec  8 02:57:16 2014
        0.00 version
           1 ordinal base
           3 number of functions
           3 number of names

    ordinal hint RVA      name

          1    0          CancelIo (forwarded to kernel32.CancelIo)
          2    1          CancelSynchronousIo (forwarded to kernel32.CancelSynchronousIo)
          3    2          GetOverlappedResultEx (forwarded to kernel32.GetOverlappedResultEx)

  Summary

        1000 .rdata
        1000 .rsrc

[jchannon]

Note: We also appear to have api-ms-win-core-io-l1-1-0.dll in our folder and I have no idea how that relates to api-ms-win-core-io-l1-1-1.dll but below is the output from api-ms-win-core-io-l1-1-0.dll in case that is useful

c:\AM 2.3 (58fad7)-windows\portal>dumpbin /exports api-ms-win-core-io-l1-1-0.dll
Microsoft (R) COFF/PE Dumper Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file api-ms-win-core-io-l1-1-0.dll

File Type: DLL

  Section contains the following exports for api-ms-win-core-io-l1-1-0.dll

    00000000 characteristics
    548513F4 time date stamp Mon Dec  8 02:59:00 2014
        0.00 version
           1 ordinal base
           7 number of functions
           7 number of names

    ordinal hint RVA      name

          1    0          CancelIoEx (forwarded to kernel32.CancelIoEx)
          2    1          CreateIoCompletionPort (forwarded to kernel32.CreateIoCompletionPort)
          3    2          DeviceIoControl (forwarded to kernel32.DeviceIoControl)
          4    3          GetOverlappedResult (forwarded to kernel32.GetOverlappedResult)
          5    4          GetQueuedCompletionStatus (forwarded to kernel32.GetQueuedCompletionStatus)
          6    5          GetQueuedCompletionStatusEx (forwarded to kernel32.GetQueuedCompletionStatusEx)
          7    6          PostQueuedCompletionStatus (forwarded to kernel32.PostQueuedCompletionStatus)

  Summary

        1000 .rdata
        1000 .rsrc

[Tratcher]

I see the same. Are API sets supposed to chain? Every doc I can find shows CancelIOEx in 1.1.1, and I haven't been able to repro this on my VMs.

[Tratcher]

I've finally been able to repro this error message by directly invoking CancelIoEx on app launch. It's still not clear why this hasn't reproduced any other way. At least the fix is easy...