ProtectedBrowserStorage throws CryptographicException occasionally when using PersistKeysToAzureBlobStorage and ProtectKeysWithAzureKeyVault

[adamashton]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Occasionally (25 times in last 30 days) users of my Blazor App are triggering an exception in our Azure hosted App Service.

The key {72bbd744-b488-0000-0000-000000000000} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning

System.Security.Cryptography.CryptographicException

Often times the app is working fine and I assume the key is being used to encrypt and decrypt the keys.

Expected Behavior

The App Service should always be able to access the key and the cryptographic exception should not occur.

Steps To Reproduce

I have followed this guide https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-7.0 and configure my Data Protection like so,

services.AddDataProtection()
  .PersistKeysToAzureBlobStorage(new Uri(options.KeyRingBlobUri), new DefaultAzureCredential())
  .ProtectKeysWithAzureKeyVault(new Uri(options.EncryptionKeyUri), new DefaultAzureCredential());

I believe my Azure access is configured correctly as it works fine for most of the time.

• App Service has role Storage Blob Data Contributor for the Blob.
• App Service has role Storage Key Vault Crypto Service Encryption User for the Encryption Key Store.
• The App is running under a Windows App Service Plan on the .NET 6.0 runtime.
• Azure.Extensions.AspNetCore.DataProtection.Blobs 1.3.2
• Azure.Extensions.AspNetCore.DataProtection.Keys 1.2.2

Exceptions (if any)

The key {72bbd744-b488-0000-0000-000000000000} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning

System.Security.Cryptography.CryptographicException:
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect (Microsoft.AspNetCore.DataProtection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect (Microsoft.AspNetCore.DataProtection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Components.Server.ProtectedBrowserStorage.ProtectedBrowserStorage.Unprotect (Microsoft.AspNetCore.Components.Server, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Components.Server.ProtectedBrowserStorage.ProtectedBrowserStorage+<GetAsync>d__8`1.MoveNext (Microsoft.AspNetCore.Components.Server, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

.NET Version

6

Anything else?

From the code in KeyRingBasedDataProtector it looks like it can access the key but the one with the specific ID is not found?

aspnetcore/src/DataProtection/DataProtection/src/KeyManagement/KeyRingBasedDataProtector.cs

Line 244 in d1f00b0

throw Error.Common_KeyNotFound(keyIdFromPayload);

[ChrisASearles]

I also get this error intermittently that I documented on StackOverflow here and cannot figure out what the problem is. Mine does cause a major issue when it occurs. Login happens fine but then I get this error from all API calls from my Blazor WASM app. I believe it has something to do with caching on the client side but cannot figure out why or how to fix it. The only way to get the app functioning again (mine runs in an Azure App Service) is to restart it and make sure the first time I hit the site comes from a private browsing session, then it works fine. If I try from a regular browser window I get the same behavior and nothing works for any users until I restart and hit immediately from a private window.

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 9 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[wzoet]

We use somewhat the same configuration in our software:

services.AddDataProtection().SetApplicationName(Configuration["DataProtectionApplicationName"]).PersistKeysToDbContext<DataProtectionKeysContext>().ProtectKeysWithAzureKeyVault(new Uri(Configuration["SomeSetting"]), new DefaultAzureCredential());

We use the SetApplicationName to support multiple deployment slots on Azure. Otherwise our keys fail every time we switch slots. Next to that, we store the keys in the database. It is the same database IDSRV uses to store user credentials, so it is reachable by the application.

At this moment we get intermittent errors as displayed above. But we suspect it has something to do with creating new keys.
We found that the key that was reported as missing was actually just inserted into the database. The creation date of the key was just moments before we saw the first errors occur in the logs.

We have no specific settings in the IDSRV options:

services.AddIdentityServer(options => { options.LicenseKey = this.Configuration["IdentityServerLicenseKey"]; options.EmitStaticAudienceClaim = true; }) .AddAspNetIdentity<ApplicationUser>()

We understand that the issue won't be resolved quickly, but it is a major issue for us. All user interaction with our system is stopped and also inter-application communication is stopped as we use IDSRV authentication there as well. Can we circumvent this for now by disabling the automatic key rotation? Or is there another way we could prevent the issue from happening?

We use .Net 8.0 and IDSRV 7.0.1
We also use IDSRV on two app-service-plan instances, we have IDistributedCache set up using the database as well.

Thanks in advance for thinking along.

[nickfortyau]

This is also a huge issue for our team. Pretty much all database access is halted until the user clears their localStorage.

[Dayana-mypos]

I have come across this problem as well. A .Net API project is using an instance of Identity Server for authentication. Between midnight and 2.a.m the users successfully create tokens, but moments later introspect returns that the tokens are inactive. There is an error logged between these 2 operations: The key {xxx} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning

[amcasey]

I'm going to close this issue. Please post additional feedback in #36157 so we can track it centrally and don't drop any. My apologies to anyone whose feedback was missed in the interim.