Do not publish expired CA certificates

[siegfriedweber]

Expired CA certificates should not be published or there should be an option to prevent this.

The secret-operator publishes all CA certificates from the Secret secret-provisioner-tls-ca into the file ca.crt of the TLS volumes.

At least OpenSearch cannot handle this:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.19.0.jar:2.19.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.19.0.jar:2.19.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:818) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:757) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:551) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:524) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:451) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.19.0.jar:2.19.0]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:809) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:757) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:551) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:524) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:451) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.19.0.jar:2.19.0]
        ... 6 more
uncaught exception in thread [main]
Caused by: org.opensearch.OpenSearchException: Invalid certificates
        at org.opensearch.security.ssl.config.KeyStoreUtils.validateKeyStoreCertificates(KeyStoreUtils.java:161) ~[?:?]
        at org.opensearch.security.ssl.config.TrustStoreConfiguration.createTrustManagerFactory(TrustStoreConfiguration.java:61) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:84) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:49) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:45) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:104) ~[?:?]
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:103) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:88) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:326) ~[?:?]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:809) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:757) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:551) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:524) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:451) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.19.0.jar:2.19.0]
        ... 6 more
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Mon Jul 28 14:14:05 UTC 2025
        at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277) ~[?:?]
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:621) ~[?:?]
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:594) ~[?:?]
        at org.opensearch.security.ssl.config.KeyStoreUtils.validateKeyStoreCertificates(KeyStoreUtils.java:147) ~[?:?]
        at org.opensearch.security.ssl.config.TrustStoreConfiguration.createTrustManagerFactory(TrustStoreConfiguration.java:61) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:84) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:49) ~[?:?]
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:45) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:104) ~[?:?]
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:103) ~[?:?]
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:88) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:326) ~[?:?]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:809) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:757) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:551) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:524) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.node.Node.<init>(Node.java:451) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.19.0.jar:2.19.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.19.0.jar:2.19.0]
        ... 6 more
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: java.security.cert.CertificateExpiredException: NotAfter: Mon Jul 28 14:14:05 UTC 2025
        at java.base/sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:621)
        at java.base/sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:594)
        at org.opensearch.security.ssl.config.KeyStoreUtils.validateKeyStoreCertificates(KeyStoreUtils.java:147)
        at org.opensearch.security.ssl.config.TrustStoreConfiguration.createTrustManagerFactory(TrustStoreConfiguration.java:61)
        at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:84)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73)
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:49)
        at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:45)
        at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:104)
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
        at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:103)
        at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:88)
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:326)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:809)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:757)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:551)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
        at org.opensearch.node.Node.<init>(Node.java:524)
        at org.opensearch.node.Node.<init>(Node.java:451)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        <<<truncated>>>

[siegfriedweber]

It should be avoided to publish an almost expired CA certificate, because the CA could expire while the Pod is starting up and when the start up is finished, the Pod sees an expired certificate and fails. It is also important to publish a non-expired CA if there could be certificates that are signed by it. To align both requirements, a threshold t must be introduced:

• A certificate may only be signed by a CA if the lifetime of the CA is longer than the certificate lifetime plus t. The next CA (e.g. 1.ca.crt instead of 0.ca.crt) must be used if the current CA does not meet this requirement.
• A CA is only published if its remaining lifetime is longer than or equal to t.

[dervoeti]

That makes sense.

I'm thinking about where to make t configurable.

It could be
a) a property of autoTls.ca
b) an annotation on the volume claim

I'm tending towards b) because it would allow to specify different thresholds for different workloads (some Pods might have much longer startup times than others).

And then make it default to 30 minutes or something.

[nightkr]

This.. seems like a bug in OpenSearch, would you expect the same thing to happen for a "static" install just because something in your system trust store expired? That seems.. wildly disproportionate.

But going beyond that for a moment, this effectively creates a "dead zone" where a certificate can be "valid", but still won't be trusted by all peers.. you'll need to adjust the logic that picks which CA should sign a newly issued certificate to make up for that dead zone.

[dervoeti]

this effectively creates a "dead zone" where a certificate can be "valid", but still won't be trusted by all peers

If I understand this correctly, this is what @siegfriedweber meant to mitigate by:

A certificate may only be signed by a CA if the lifetime of the CA is longer than the certificate lifetime plus t

In other words: The maximum certificate lifetime is ca_lifetime minus t.

I do agree that be behavior of OpenSearch seems weird and it it's a bit awkward to make this rather complicated workaround. But there's probably no other way to solve it if we want to support OpenSearch, I'll briefly look into possible alternative solutions though. Currently I'm at least not aware of any other products failing because of this.

[nightkr]

Right, yeah.