opensearch-project
Security TLS configuration fails
What is the bug?
When I enable tls, I have decided to use self signed certificates using the opensearch cluster CA for the transport layer. However I want to create a custom certificate for the http layer. I am using this configuration:
`security:
config: # Everything related to the securityconfig
securityConfigSecret:
name: ${security_config_secret}
adminCredentialsSecret:
name: ${admin_credentials_secret}
tls:
transport:
generate: true
perNode: true
http:
generate: false # --> here I disable certificate generation
secret:
name: ${tls_rest_secret_name} # Name of the secret that contains the provided certificate`
However, the opensearch-cluster-bootstrap fails with this error:
init-sysctl vm.max_map_count = 262144
opensearch [2023-12-23T13:50:23,243][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-bootstrap-0] SSL dual mode is disabled
opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-bootstrap-0] OpenSearch Config path is /usr/share/opensearch/config
opensearch [2023-12-23T13:50:23,501][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] JVM supports TLSv1.3
opensearch [2023-12-23T13:50:23,503][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
opensearch [2023-12-23T13:50:23,555][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-bootstrap-0] uncaught exception in thread [main]
opensearch org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.3.0.jar:2.3.0]
opensearch Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.
How can one reproduce the bug?
Using the security setup above
What is the expected behavior?
The startup process should run successfully
Do you have any additional context?
I am using the opensearch-cluster helm chart to apply this configuration. The values above correspond to the values.yaml used.
According to the documentation, It supposed that the tls configuraton for transport and http could be configured in an independent way. However, I wonder either if once you disable the generation for any of them we need to use our own CA for both of them. or if there is a missing setup that is not documented.
On the other hand. If I set the http > generate parameter to true
http: generate: true
I don't get any error, and the cluster start up process is executed successfully.
I appreciate your help.
Cheers.
Hi, looking at the opensearch cluster chart. I noticed that this line could be the reason of the problem
https://github.com/opensearch-project/opensearch-k8s-operator/blob/f88d40d8ae679cb400826bffc4d7939db42edff0/charts/opensearch-cluster/templates/opensearch-cluster-cr.yaml#L284
As you can see there the transport.generate value is dependen of the http.generate value and it shouldn't be. Is there someone working on this?
Hi I made the change above on the chart locally and it seems to work. However, I get a new error on the opensearch-cluster-securityconfig-update pod:
** This tool will be deprecated in the next major release of OpenSearch ** ** https://github.com/opensearch-project/security/issues/1755 **
Will connect to opensearch-cluster.opensearch-cluster.svc.cluster.local:9200 ... done ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetSecurity Admin v7 See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting. Trace: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting. at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947) at org.opensearch.client.RestClient.performRequest(RestClient.java:332) at org.opensearch.client.RestClient.performRequest(RestClient.java:320) at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462) at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159) Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288) at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356) at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ... 19 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 24 more
Is there anyone working on this?. Let me know if you need more info to continue.
Cheers.
@ericklife28 have you managed to find a solution? Facing the same issue with Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error ((
Hi @saketmht @stefan-heilig-mw i think the root cause is here, you are using the node transport CA
https://github.com/opensearch-project/opensearch-k8s-operator/blob/2e78967d3d5868d719442574f5d26ee8fae4a17a/opensearch-operator/pkg/reconcilers/securityconfig.go#L30
https://github.com/opensearch-project/opensearch-k8s-operator/blob/2e78967d3d5868d719442574f5d26ee8fae4a17a/opensearch-operator/pkg/reconcilers/securityconfig.go#L161
when user have a custom CA for http under /usr/share/opensearch/config/tls-http/ca.crt we should honor and trust this cert
as per this doc https://github.com/opensearch-project/opensearch-k8s-operator/blob/2e78967d3d5868d719442574f5d26ee8fae4a17a/charts/opensearch-operator/files/opensearch.opster.io_opensearchclusters.yaml#L5355
