HTTP Error 500.30 - ANCM In-Process Start Failure

[gthvidsten]

Yet another HTTP Error 500.30 - ANCM In-Process Start Failure :S

Describe the bug

I have a .NET Core 3 app that runs just fine locally, so I deploy it to an Azure App Service, where it also runs just fine.
Now that we've tested it I want to deploy it to another Azure App Service for more long term testing. This App Service has (seemingly) the exact same settings as the first one.

On the second App Service, I get HTTP Error 500.30 - ANCM In-Process Start Failure

In the Application Event Logs there is only this error:

Application '/LM/W3SVC/870738454/ROOT' with physical root 'D:\home\site\wwwroot' failed to load coreclr. Exception message:
CLR worker thread exited prematurely
Process Id: 19728.
File Version: 13.0.19309.1. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 22dedcb

Based on other 500.30 issue reports I have made absolutely sure that my .NET Core 3 app does not reference any .NET Core 2 packages (If it had, the first App Service should have failed as well, but it works there).

I tried editing the web.config in the App Service to enable stdout-logging, but because I have WEBSITE_RUN_FROM_PACKAGE = 1 I'm not allowed to edit the web.config file. If I change that value to 0 the files in D:\home\site\wwwroot disappears.

When trying to start the .NET Core 3 app from the Kudu Console, nothing happens. No output, no anything. It just seems to load forever. Both dotnet MyApp.dll and MyApp.exe gives the same result.

So why does this happen? How can I fix this?

To Reproduce

Seeing as this happens in one App Service, and not another, yet these two App Services should have the same configuration, I don't know how to specifically reproduce this.

Further technical details

ASP.NET Core version: 3.0

dotnet --info (ran from the Kudu console):

D:\home>dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.109
 Commit:    586f23c400

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x86
 Base Path:   D:\Program Files (x86)\dotnet\sdk\2.2.109\

Host (useful for support):
  Version: 3.1.0
  Commit:  65f04fb6db

.NET Core SDKs installed:
  1.1.14 [D:\Program Files (x86)\dotnet\sdk]
  2.1.509 [D:\Program Files (x86)\dotnet\sdk]
  2.2.109 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.13 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.13 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.16 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.14 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Running in Azure App Service, not an IDE.

[jkotalik]

I tried editing the web.config in the App Service to enable stdout-logging, but because I have WEBSITE_RUN_FROM_PACKAGE = 1 I'm not allowed to edit the web.config file. If I change that value to 0 the files in D:\home\site\wwwroot disappears.

I'm not familiar with WEBSITE_RUN_FROM_PACKAGE, however it may be worth trying to get the application outside of the package to get logs.

When trying to start the .NET Core 3 app from the Kudu Console, nothing happens. No output, no anything. It just seems to load forever. Both dotnet MyApp.dll and MyApp.exe gives the same result.

Very weird. That should allow the application to run (even though it isn't exposed). Can you try to deploy application as standalone on that instance?

[gthvidsten]

The "Azure App Service Deploy" task in DevOps always deploys as a package, not as a standalone, and there are no settings in it to change this behaviour.

I also tried downloading the publish profile from the App Service, imported it into Visual Studio and tried publishing from Visual Studio, but that just gives an error, so I'm afraid I'm not able to deploy as standalone.

------ Publish started: Project: Web, Configuration: Release Any CPU ------
Web -> C:\MyProject\bin\Release\netcoreapp3.0\MyProject.dll
Web -> C:\PMyProject\bin\Release\netcoreapp3.0\MyProject.Views.dll
XCOPY "C:\MyProject\appsettings.Release.json" "C:\MyProject\appsettings.Development.json*" /q /y
1 File(s) copied
Web -> C:\MyProject\obj\Release\netcoreapp3.0\PubTmp\Out
C:\Program Files\dotnet\sdk\3.1.100\Sdks\Microsoft.NET.Sdk.Publish\targets\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(140,5): Error : Web deployment task failed. ((13.01.2020 10:11:54) An error occurred when the request was processed on the remote computer.)

(13.01.2020 10:11:54) An error occurred when the request was processed on the remote computer.
Could not find file 'D:\home\site\wwwroot\App_Offline.htm'.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
at System.Xml.XmlWriterSettings.CreateWriter(String outputFileName)
at Microsoft.Web.Deployment.AppOfflineRuleHandler.AddAppOfflineFilesToEachApp(DeploymentBaseContext baseContext, Boolean whatIf)
at Microsoft.Web.Deployment.AppOfflineRuleHandler.Update(DeploymentSyncContext syncContext, DeploymentObject destinationObject, DeploymentObject& sourceObject, Boolean& proceed)
at Microsoft.Web.Deployment.DeploymentSyncContext.HandleUpdate(DeploymentObject destObject, DeploymentObject sourceObject)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncDirPathChildren(DeploymentObject destRoot, DeploymentObject sourceRoot)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildren(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenNoOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildren(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.SyncChildrenOrder(DeploymentObject dest, DeploymentObject source)
at Microsoft.Web.Deployment.DeploymentSyncContext.ProcessSync(DeploymentObject destinationObject, DeploymentObject sourceObject)
at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObject destObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable1 syncPassId, String syncSessionId) at Microsoft.Web.Deployment.DeploymentAgent.HandleSync(DeploymentAgentAsyncData asyncData, Nullable1 passId, String user, String siteName)
Publish failed to deploy.

[gthvidsten]

Additional info:

The Visual Studio publish issue was because the App Service had an app setting WEBSITE_RUN_FROM_PACKAGE = 1. I removed this app setting and I could publish from Visual Studio.

I published from Visual Studio using "Web Deploy" instead of "Web Deploy Package", but still the App Service gave the HTTP Error 500.30 - ANCM In-Process Start Failure error.

The Microsoft documentation suggest to enable "stdout logging", but setting stdoutLogEnabled="true" (in D:\home\site\wwwroot\web.config on the AppService) did not create any log files (I also tried manually creating the output folder, but still no logs.

As a final desperate attempt to get the log files to appear I removed hostingModel="inprocess" from the same web.config file. Now, my App Service started and works as it should!

This is not good enough, of course, because as soon as I deploy from DevOps again it is back to "inprocess" hosting model and the App Service fails with the initial error.

So something is obviously wrong, I can't get any information about why things are wrong, but not running the App Service "inprocess" causes everything to work!

[jkotalik]

One thing that has been coming up recently is a regression in the platform here: #17973. Potentially this is the issue you are hitting, as people have been showing the same symptoms that you are seeing.

If that doesn't fix it, inside of your csproj for your side, you can add <AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel> and that should force your side to publish out of process.

[analogrelay]

@gthvidsten can you try applying the workarounds listed in #17973 and see if that fixes the issue? If so, we expect this to be fixed in an upcoming patch. If not, let us know here and we can investigate further.

[gthvidsten]

After a lot of troubleshooting with one of our sysadmins, we finally found the problem, and it is the one described here: dotnet/AspNetCore.Docs#10725

That issue is closed, but I would say it was not fixed as I never found the mention of Extensions in documentation regarding the 500.30 ANCM In-Process Start Failure. If that issue was supposed to fix the 500.30 error, then that isn't fixed either.

We removed the extensions from the AppService, and everything started working. (These were ASP.Net Core 2.2 preview extensions, but we're using ASP.Net Core 3.0, so why they caused conflicts at all is beyond me).

[analogrelay]

Triage: There can be conflicts when you have a preview site extension installed, since it overrides the entire .NET Core runtime in use, but we think the host should be searching through both the site extension and machine-level runtimes to find a suitable runtime so it shouldn't be causing this problem. We'll take a look and see if we can repro this.

The Site Extension also brings a different (older) version of ANCM but we think that shouldn't conflict.

It's definitely still a good idea to uninstall old preview extensions, but we'll take a look and see if there's an issue here.