Kestrel + IIS is falling over with a small load in production with RTM

[jruckert]

We are currently experiencing an issue in production (after upgrading from DNX to RTM) when our Web API is

• loaded with around 10 req/second
• Some calls are taking > 2 second with sometimes over 1mb in a payload.
• Our average API timing and payload would be < 1 second and around 30kb of data.

It seems that sometimes when the larger API get queued whilst processing the smaller requests it overwhelms Kestrel.

We have a setup as follows:

• Dotnet Core RTM (1.0.0)
• Webhosting pack (1.0.0) installed
• Windows Server 2012 R2
• Large number of API's
• HTTPS
• Windows Authentication
• Entity Framework 7
• SQL Server 2014 (different server)

What we find is that we start eventually getting 502.3 errors and then Kestrel become unresponsive or takes a very long time to respond (e.g. > 30 seconds for a specific API when usually the API would respond in less than a second).

The only way to resolve this issue is to recycle the app pool and kill the process thread (currently we are having to do this at least once a day in production).

Do we need to tune IIS to handle this scenario and how can we diagnose, provide log files to get to the bottom of this, as prior to RC2 (we upgraded from DNX through to RTM) the website has never crashed.

[jruckert]

We are also planning on running the load tool wrk for a few larger API's > 2 seconds and large JSON response payload to see if we can replicate this issue if this would help.

[jruckert]

From the IIS Performance Counters:

5:20pm - thread count started growing dramatically
5:40pm - access issues to website first reported
5:50pm - IIS Application Pool stopped and started, and thread count drops

Process counters for w… The this is IIS w3wp worker process.

Process counters for P… This is the NetCore process.

[wpostma]

I am having a similar problem. Nowhere near that level of load. 3 tiny dotnet apps, on a server that also has an SQL Server 2016 instance.

I get about 6 seconds of 404 errors, followed by a permanent 502.5 condition, that lasts until I do a stop on w3svc and then start it again. Restarting app pools is not sufficient.

http://stackoverflow.com/questions/38613615/dotnet-process-inside-iis-crashes-how-can-i-determine-cause-and-remediate/38745424#38745424

Do you get anything interesting if you enable stdout logging?

What do you see in the iis text log files (c:\inetpub\logs)? Is it similar to the log output in my answer in stackoverflow link above?

[KLuuKer]

Looks like i may also have the same issue,
we finished converting our multi-tenant framework 4.6 mvc 5 to core mvc 1
our traffic monthly is about:

• unique visitors 250.000
• uncached requests 1.300.000

Our setup:

• Dotnet Core RTM (1.0.0) running on top of fullframework
• Webhosting pack (1.0.0) installed
• Windows Server 2012 R2 (all patches installed)
• Single Application Pool just for kestrel
• HTTP (tls is getting offloaded on a different machine)
• Anonymous authentication (public facing site)
• NO entity framework (this is the frontend, the backend exposes data via json\rest)
• SQL Server 2016 (different machine)

dotnet --version

Microsoft .NET Core Shared Framework Host

  Version  : 1.0.1
  Build    : cee57bf6c981237d80aa1631cfe83cb9ba329f12

stdout

Hosting environment: Production
Content root path: C:\inetpub\project.name-mvccore
Now listening on: http://localhost:21704
Application started. Press Ctrl+C to shut down.

and then all loads of variations of this that never get finished (all from real users hitting our real e-commerce site)

info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://example.com/variouspaths/that-should-work/

i also ran it with debug logging, no real difference (i can give it to the microsoft team if they want it)

{
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Debug",
      "System": "Debug",
      "Microsoft": "Debug"
    }
  }
}

[KLuuKer]

@shanselman @DamianEdwards
sorry to @ you like this, but we are letting people hit our old mvc 5 app until this can be fixed
and so we cant deploy new code in the mean time (saas e-commerce app and our customers are getting angry)

[muratg]

cc @sajayantony @sivagms

[muratg]

@KLuuKer Could you share the logs?

[pan-wang]

Could anyone provide a repro so that I can take a look at it?
404 error is generated by Krestral.
502 means IIS could not start the Krestral process. For this error, please take a look at windows event log. It usually gives some error code indicating the root cause of starting failure.
Please also try enabling IIS FREB log which might help to identify the root cause.

[KLuuKer]

@muratg can you give me your email here or dm via twitter to https://twitter.com/KLuuKer

[wpostma]

I have set up an indentical system to the one that is falling over on my W2012R2 vm, on an azure vm, but it's using Windows Server 2016 TP5 on the azure VM, and if it reproduces for me, I'll get access to any MS employee who needs it. Right now I'm running exactly the same setup as the local VM that failed, except without using the IIS URL Rewrite extension. If no reproduce, I'll add back in IIS URL rewrite. I suspect that the problem is exacerbated or made more likely by me using three apps in a single pool.

[wpostma]

@pan-wang How do I enable IIS FREB for this context? I don't know what to enter for "Specify Content to Trace" prompt in iismgr. All contents? Then 502 error?

[KLuuKer]

Just making this clear: i am running kestrel\mvc core 1 using the full framework 4.6 because of legacy bits

[jruckert]

Also the spin up time for our process due to the large dependency injection requirement is approx 5 seconds before IIS can access it. Would this affect things?

[pan-wang]

IIS will log the start and failure of Kestrel process to windows event log. In case of 502 error, this log will be most important source to help identifying the root cause of the failure. If possible, please do take a look at it and share it.

@jruckert IIS has a timeout setting on Kestrel process starting, default is 2 minutes.
@wpostma Yes, you can choose all contents and your current FREB setting looks good. For your reference, current hosting mode is one Kestrel application per IIS apppool (site). There will be some issues especially about recycling/file change if you host multiple apps together. I am working on changing it to allow user to host multiple applications inside one apppool.

[guardrex]

Ping @tuespetre ... and I ... have an interest in your (@pan-wang) comment on multiple apps per pool, as @tuespetre is doing it in staging, and the green dino here is 🐔 too chicken 🐔 to try it at all. 😄

I am working on changing it to allow user to host multiple applications inside one apppool.

@pan-wang, there is no GH issue we can track on for that, correct?