[BUG] OpenID connection with certificate verification from keystore.

[GypsyJR777]

What is the bug? An error occurs when using OpenID and PKCS12:

[2024-02-20T10:29:39,964][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [node-01] Error creating JWT authenticator. JWT authentication will not work
com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:337) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:195) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:116) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:130) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) [opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) [opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:310) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:87) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:281) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:395) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:379) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$new$0(ConfigurationRepository.java:221) [opensearch-security-1.3.0.jar:1.3.0]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Keystore is closed
        at org.opensearch.common.settings.KeyStoreWrapper.ensureOpen(KeyStoreWrapper.java:672) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:593) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:204) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:194) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting.get(SecureSetting.java:116) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:96) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:92) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:341) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 22 more
[2024-02-20T10:29:39,972][WARN ][o.o.s.s.ReflectionHelper ] [node-01] Unable to enable 'com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-02-20T10:29:39,980][ERROR][o.o.s.s.DynamicConfigModelV7] [node-01] Unable to initialize auth domain openid_auth_domain=AuthcDomain [http_enabled=true, transport_enabled=true, order=0, http_authenticator=HttpAuthenticator [challenge=false, type=openid, config={openid_connect_url=my_url, openid_connect_idp={enable_ssl=true, verify_hostnames=false}, jwks_uri=my_uri, subject_key=preferred_username, roles_key=realm_access, roles_sub_key=roles}], authentication_backend=AuthcBackend [type=noop, config={}], description=Authenticate via proxy] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12]; nested: SSLConfigException[Error loading trust store from /etc/opensearch/certs/self/node.p12]; nested: IllegalStateException[Keystore is closed];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:406) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:310) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:87) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:281) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:395) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:379) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$new$0(ConfigurationRepository.java:221) [opensearch-security-1.3.0.jar:1.3.0]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: java.lang.RuntimeException: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:85) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:337) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:195) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:116) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:130) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: java.lang.IllegalStateException: Keystore is closed
        at org.opensearch.common.settings.KeyStoreWrapper.ensureOpen(KeyStoreWrapper.java:672) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:593) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:204) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:194) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting.get(SecureSetting.java:116) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:96) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:92) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:341) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:200) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:120) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:134) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more

How can one reproduce the bug? Steps to reproduce the behavior:

1. Take some server
2. Create security config like this:

---
_meta:
  config_version: 2
  type: config

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      openid_auth_domain:
        description: Authenticate via proxy
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: openid
          challenge: false
          config:
            openid_connect_url: my_url
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
            jwks_uri: my_uri
            subject_key: preferred_username
            roles_key: realm_access
            roles_sub_key: roles
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

3. Paste the excerpt into opensearch.yml:

plugins.security.ssl.transport:
  enabled: true
  keystore_type: PKCS12
  truststore_type: PKCS12
  keystore_filepath: "/etc/opensearch/certs/self/node.p12"
  truststore_filepath: "/etc/opensearch/certs/self/node.p12"
  enabled_protocols: ["TLSv1.2", "TLSv1.3"]
  enabled_ciphers: ["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]
  enforce_hostname_verification: true

4. Start opensearch
5. See logs

What is the expected behavior? Either do not allow work with keystore at all, or get the password and continue working without errors.

What is your host/environment?

• OS: RHEL 8.1
• Version opensearch 2.11.0.0
• Plugins: default

Do you have any screenshots? Nothing

Do you have any additional context? No

[derek-ho]

[Triage] This seems like a bug and I am seeing OpenSearch 1.3 in the stacktrace. We would want to take a quick look at this and see if there is a bug that needs fixing.

[GypsyJR777]

Version 1.3 is the versioning that I gave to my build. In fact, version 2.11 is used.