[WebToolsE2E] Run .exe/.dll file with error "An exception occurred while trying to decrypt the element." from publish folder after publishing .NET 9.0 Razor/MVC/Blazor web app project.

[v-doyang]

REGRESSION INFO: Worked in .NET 8.0
Azure DevOps work item: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/2183540

Testcases Affected
AspNetCore90_RazorPages_IndAuth_ScaffoldingAsync
AspNetCore90_MVC_IndAuth_CreateRunPublishAsync
AspNetCore90_Blazor_WebApp_IndAuth_Scaffolding_Async
AspNetCore90_EmptyTemplate_HotReload
AspNetCore90_BlazorWebAssemblyStandaloneApp_NoAuth_IncludeSamplePages_ProgressiveWebApplication_CreateRunPublishAsync
AspNetCore90_Blazor_WebApp_IndAuth_CreateRunPublishAsync
AspNetCore90_RazorPages_IndAuth_CreateRunPublishAsync

Platform

• Windows
• macOS
• Linux

INSTALL STEPS

1. Clean machine: Win 11 x64 22h2 ENU
2. Install VS Dev17.12 Preview 1 build main-35201.144. Branch channel

• Web workload
• Additional project templates(preview)
• Run: \vspreinstall\preinstall\preinstall.cmd

Repro Steps

1. Create a new project > ASP.NET Core Web App (Razor Page) > .NET 9.0 > Create.
2. Publish to Folder and open output folder. Run .exe/.dll file.

Note
This issue only repro on the Pipeline machine when run our automation tests.

Error log:
ErrorInfo.txt

Actual Result
There are some errors after run .exe file

Expected Result
There is no error/warning after run .exe file.

Error message:
Error message .txt

[javiercn]

@v-doyang thanks for the report.

@amcasey does this look familiar to you? Have we made any change in this are that might impact this?

[amcasey]

I made a bunch of Data Protection changes (with appcontext switches, if we want to experiment), but that was in the spring. Most data protection errors (including the ones targeted by my changes) are about missing keys, rather than keys that are present but unable to be processed.

It's not impossible that my cleanup turned a swallowed exception into a rethrown exception, but it's not immediately obvious to me why we'd want to swallow an exception like this.

@v-doyang Does it happen on every run or only occasionally/once? When was the test last run successfully?

[balachir]

@amcasey this issue was found by our automated runs for VS 17.12 P1.

Here is what I understood from my validation team when I discussed this issue with them earlier today.

• If they manually set up a VM in Azure DTL with a clean install of VS 17.12 P1 (includes SDK 9.0 P6 at the moment) and try the repro steps mentioned in this issue, they cannot reproduce it.
• When a machine gets setup as part of our automated pipeline runs with a clean install of VS 17.12 P1 (includes SDK 9.0 P6 at the moment) and our automated tests run, they can reproduce this issue. If they then login to this machine and manually test the scenario, they can reproduce it as well.

I believe they selected the same options during VS install (i.e. default install of Web workload). I'm not sure what's the difference between the two types of machines and why only the latter one would give these data protection errors. Any suggestions on how they can investigate this further? If you need a repro machine to investigate, let us know and we can provide you with one offline.

cc: @v-doyang @javiercn

[amcasey]

@balachir Thanks for the details! When would the preceding run have been? Is the baseline SDK 8.0 or a previous preview of 9.0?

Can I get access to one of the failing VMs?

Edit: I missed this "If you need a repro machine to investigate, let us know and we can provide you with one offline.". Yes, please. 😄

[amcasey]

@balachir @v-doyang Another thing it would be helpful to test is whether adding this at the beginning of the app code changes the behavior.

AppContext.SetSwitch("Microsoft.AspNetCore.DataProtection.KeyManagement.DisableAsyncKeyRingUpdate", true);

I don't think it will, but that will disable the scariest data protection change made in 9.0 and it would be nice to rule it out.

[balachir]

@amcasey I believe this is showing up now because the 9.0 SDK was included as the default SDK recently in VS 17.12 P1 and hence our automated tests are running against 9.0 target framework now in addition to 8.0. Unfortunately, I don't know how to get a failing VM right now. I'll work with @v-doyang when she's back on Monday and we'll get you one.

[amcasey]

It didn't make it into the screenshot, but this is interesting and relevant because it is new:

warn: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver[12]
      Key {33b89a5b-93d6-4564-b14e-ed5325af347e} is ineligible to be the default key because its CreateEncryptor method failed after the maximum number of retries.
      System.AggregateException: One or more errors occurred. (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.) (Error occurred during a cryptographic operation.)
       ---> System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
         at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.get_Descriptor()
         at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.CreateEncryptor()
         at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key, Int32& retriesRemaining)
         --- End of inner exception stack trace ---
       ---> (Inner Exception #1) System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
         at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapiCore(Byte* pbProtectedData, UInt32 cbProtectedData, Byte* pbOptionalEntropy, UInt32 cbOptionalEntropy)
         at Microsoft.AspNetCore.DataProtection.Cng.DpapiSecretSerializerHelper.UnprotectWithDpapi(Byte[] protectedSecret)
         at Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor.Decrypt(XElement encryptedElement)
         at Microsoft.AspNetCore.DataProtection.XmlEncryption.DpapiXmlDecryptor.Decrypt(XElement encryptedElement)
         at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.get_Descriptor()
         at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.get_Descriptor()
         at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.CreateEncryptor()
         at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key, Int32& retriesRemaining)<---

The retries don't really make sense when using DPAPI (rather than, e.g. AKV), but they're relatively harmless. My immediate hypothesis is that this has always failed but was formerly swallowed. The new warning is intentional, but I'm guessing the failures should still be swallowed.

Edit: this is the relevant PR. It looks like it does what it did before, but in a loop, so I think the only meaningful change is that we got 10 failure messages, rather than 1. Swallowing or not swallowing the exception doesn't seem like it would make a difference because the failure log message is produced so close to where the exception is thrown.

[amcasey]

@BrennanConroy Is this one of the data protection errors we were seeing in the MVC tests? I couldn't find it in the chat, but I think I remember talking about it.

[BrennanConroy]

Key {33b89a5b-93d6-4564-b14e-ed5325af347e} is ineligible to be the default key because its CreateEncryptor method failed after the maximum number of retries.

Don't remember seeing this one.

[amcasey]

It's not immediately obvious how it could be related, but here's a Data Protection change in 9.0 I didn't know about: #51253

[amcasey]

There are subtle changes to this logic that may result in a different key being picked than formerly (though I can't see how, on a fresh VM). If that key were also undecryptable, we could start seeing this issue.

https://github.com/dotnet/aspnetcore/blob/1d88c6cf5e0b62050f4b17431b9a34fa9e26070a/src/DataProtection/DataProtection/src/KeyManagement/DefaultKeyResolver.cs#L163-169