AddDataProtection().PersistKeysToFileSystem() used with pods causes race condition and two data-protection keys to be generated at the same time.

[Duke4848]

Describe the bug

I use

 services.AddDataProtection().PersistKeysToFileSystem(new DirectoryInfo(filesharePath);

to generate data-protection keys. This produces the issue on initial application startup and possibly the same may happen during rotation of data-protection key. What essentially happens is:

1. 2 or more instances of app on startup detect that there is no data-protection key in the fileshare
2. 2 or more instances generate the key thus the pods have different data-protection keys for next 24 hours unless I force pods to restart. This causes for example issues with antiforgery tokens not being accepted by instances running on other pods.

The same may probably happen when the existing data-protection key is near expiration time and 2 instances running on separate pods decide do generate the new data-protection key.

Is it possible to do something about it?

To Reproduce

Use

 services.AddDataProtection().PersistKeysToFileSystem(new DirectoryInfo(filesharePath);

and deploy your ASP.NET Core app two few pods.

Exceptions (if any)

Further technical details

• ASP.NET Core version 3.1

[blowdart]

You could stagger the start, but right now that's your only option I'm afraid.

[Duke4848]

You could stagger the start, but right now that's your only option I'm afraid.

What is correct technique to achieve that? Should it be done somehow on the application level or pod level? But still, would that happen later on, when the ASP .NET Core app creates new data-protection key because of expiration of the old key? Then I think no staggering would help as app is in running state already on all pods.

[blowdart]

Linking to earlier #3514

[blowdart]

Closing as dupe (well, will be fixed as part of #3514)

[blowdart]

@Duke4848 We're discussing this in depth today, Are you actually seeing problems here using the file system? We have a mechanism in place for this, where for 5 minutes after a key is created the machine that created the key will refresh the keyring if it gets a key it doesn't understand. When the keyring is refresh the system will switch to using the earliest key possible that is still valid.

If you can turn on verbose logging you will see the keys identifiers data protection is reading, and which one is chosen. Those logs would help us track down what is happening.