security icon indicating copy to clipboard operation
security copied to clipboard
verified publisher icon opensearch-project

[BUG] Exception when reloading self signed certs via REST API

Open reneradoi opened this issue 1 year ago • 6 comments

What is the bug? As part of the TLS cert renewal process I want to use the REST API for hot reload of transport and http certificates. When reloading the already used certificates, everything works fine (response code 200). But when I want to apply newly generated, self signed certificates, I always run into the same exception telling me that new certs do not have valid Issuer DN, Subject DN or SAN.

I’ve checked my configuration and the validity of the certs, everything seems to be fine here. If the application is restarted, the new certs are also picked up without any issue. Only reloading via the API produces this error.

How can one reproduce the bug? Steps to reproduce the behavior:

  • Configure and start OpenSearch with self signed certificates as in Generate Certificates, but with a list of multiple entries with DNSName in the Subject Alternative Names section
  • after the application has settled, generate new self signed certificates (the same way like before, with exactly the same Issuer, Subject and Subject Alternative Names
  • try to upload the new certificate via API call to _plugins/_security/api/ssl/transport/reloadcerts

What is the expected behavior?

  • Validation of the new certificate should be ok because Issuer, Subject and SANs are the same
  • new certificate should be accepted and API call should return 200

What is your host/environment?

  • OS: Ubuntu 22.04
  • Version: Opensearch 2.14.0
  • Plugins:

Do you have any screenshots? API response:

{"error":{"root_cause":[{"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];"}],"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];","caused_by":{"type":"security_exception","reason":"Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.","caused_by":{"type":"exception","reason":"New Certs do not have valid Issuer DN, Subject DN or SAN."}}},"status":500}

Server logfile:

[2024-06-21T09:04:26,877][WARN ][r.suppressed             ] [opensearch-30] path: /_plugins/_security/api/ssl/transport/reloadcerts, params: {certType=transport}
java.io.IOException: OpenSearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:203) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$3(SecuritySSLCertsApiAction.java:117) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.validation.ValidationResult.valid(ValidationResult.java:87) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$5(SecuritySSLCertsApiAction.java:105) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.RequestHandler$RequestHandlersBuilder.lambda$add$2(RequestHandler.java:97) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.AbstractApiAction.lambda$prepareRequest$23(AbstractApiAction.java:614) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.14.0.jar:2.14.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:484) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more
Caused by: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.validateNewCerts(DefaultSecurityKeyStore.java:644) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:462) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more

Do you have any additional context? Relevant part of the config file opensearch.yml:

plugins.security.authcz.admin_dn:
- O=opensearch-x7ga,CN=admin
plugins.security.ssl.transport.pemcert_filepath: certificates/unit-transport.cert
plugins.security.ssl.transport.pemkey_filepath: certificates/unit-transport.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.pemcert_filepath: certificates/unit-http.cert
plugins.security.ssl.http.pemkey_filepath: certificates/unit-http.key
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.clientauth_mode: OPTIONAL
cluster.name: opensearch-x7ga
node.name: opensearch-30
node.roles:
- data
- ingest
- ml
- coordinating_only
- cluster_manager
discovery.seed_providers: file
plugins.security.disabled: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.restapi.roles_enabled:
- all_access
- security_rest_api_access
plugins.security.unsupported.restapi.allow_securityconfig_modification: true
plugins.security.ssl_cert_reload_enabled: true

I did a lot of debugging on this, output (incl. the current and new cert) can be found here. The essential part is this: When comparing the SAN lists of the current and the new cert, the maps are being sorted. This sort function breaks:

opensearch[opensearch-31][generic][T#11][1] print currentCertDNList
 currentCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [2, opensearch-31], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"
opensearch[opensearch-31][generic][T#11][1] print newCertDNList
 newCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, opensearch-31], [2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"

It's happening an this position in the source code: https://github.com/opensearch-project/security/blob/3dae7a75692b058a460aa2a9dda1d0aa5faae0ef/src/main/java/org/opensearch/security/ssl/DefaultSecurityKeyStore.java#L675-L679

reneradoi avatar Jun 21 '24 15:06 reneradoi

Thank you @reneradoi, it is known issue. We are planning to add https://github.com/opensearch-project/security/issues/4427 as a first step to fix hot reload problem.

willyborankin avatar Jun 21 '24 18:06 willyborankin

Hi @willyborankin I'm not sure if it's the same issue. The feature request you linked is about the ssl reload only being performed on the corrdinator node instead of the whole cluster, but in this issue I'm encountering the certs cannot be reloaded at all, also not to a single node.

As far as I've debugged this is because of a sorting operation when validating the new cert against the current one (reg. issuer, subject and sans).

reneradoi avatar Jun 24 '24 09:06 reneradoi

Hi @reneradoi, agree my comment is misleading sorry about that.

willyborankin avatar Jun 24 '24 11:06 willyborankin

[Triage] Hi @reneradoi thank you for filing this issue. Looks like a real issue, and something that should be fixed. Going to mark as triaged.

stephen-crawford avatar Jun 24 '24 15:06 stephen-crawford

I'm working on hot reload, so will try to fix this one as well.

willyborankin avatar Jun 24 '24 16:06 willyborankin

I have the same error on OpenSearch 2.8.0 with Lets Encrypt certificates. The issuer CA is different in the new certificate and reload doesn't work

[2024-07-08T13:58:19,984][ERROR][o.o.s.d.r.a.SecuritySSLCertsAction] [es-logs-5.srvroom.domain.net] Reload of certificates for transport failed
org.opensearch.OpenSearchSecurityException: Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:535) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]
at org.opensearch.security.dlic.rest.api.SecuritySSLCertsAction.handlePut(SecuritySSLCertsAction.java:238) [opensearch-security-2.8.0.0.jar:2.8.0.0]
at org.opensearch.security.dlic.rest.api.SecuritySSLCertsAction.handleApiRequest(SecuritySSLCertsAction.java:135) [opensearch-security-2.8.0.0.jar:2.8.0.0]
at org.opensearch.security.dlic.rest.api.AbstractApiAction.lambda$prepareRequest$2(AbstractApiAction.java:414) [opensearch-security-2.8.0.0.jar:2.8.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
at org.opensearch.security.ssl.DefaultSecurityKeyStore.validateNewCerts(DefaultSecurityKeyStore.java:572) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:524) ~[opensearch-security-2.8.0.0.jar:2.8.0.0]
... 9 more
[2024-07-08T13:58:19,981][INFO ][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [es-logs-5.srvroom.domain.net] User User [name=CN=es-logs-admin.srvroom.domain.net, backend_roles=[], request

olddanmer avatar Jul 12 '24 07:07 olddanmer