[BUG] opensearch - org.opensearch.security.OpenSearchSecurityPlugin - fail to load class

[pedrocassalpacheco]

The pods fail to start with the following exception:

Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init), configfile (init) Enabling OpenSearch Security Plugin Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. If a password is not provided, the setup will quit. For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/

OpenSearch Security Demo Installer

** Warning: Do not use on production or public reachable systems **

OpenSearch install type: rpm/deb on Linux 6.1.100+ amd64 OpenSearch config dir: /usr/share/opensearch/config/ OpenSearch config file: /usr/share/opensearch/config/opensearch.yml OpenSearch bin dir: /usr/share/opensearch/bin/ OpenSearch plugins dir: /usr/share/opensearch/plugins/ OpenSearch lib dir: /usr/share/opensearch/lib/ Detected OpenSearch Version: 2.17.1 Detected OpenSearch Security Version: 2.17.1.0 /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit. Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin WARNING: Using incubator modules: jdk.incubator.vector WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release Oct 21, 2024 9:02:15 PM sun.util.locale.provider.LocaleProviderAdapter WARNING: COMPAT locale provider will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-10-21T21:02:15,734][INFO ][o.o.n.Node ] [opensearch-cluster-master-1] version[2.17.1], pid[1], build[tar/1893d20797e30110e5877170e44d42275ce5951e/2024-09-26T21:59:32.078798875Z], OS[Linux/6.1.100+/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.4/21.0.4+7-LTS] [2024-10-21T21:02:15,736][INFO ][o.o.n.Node ] [opensearch-cluster-master-1] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true] [2024-10-21T21:02:15,736][INFO ][o.o.n.Node ] [opensearch-cluster-master-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11737335039693201605, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-10-21T21:02:15,920][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [opensearch-cluster-master-1] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled [2024-10-21T21:02:16,628][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-1] SSL dual mode is disabled [2024-10-21T21:02:16,628][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-1] OpenSearch Config path is /usr/share/opensearch/config [2024-10-21T21:02:16,831][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-1] JVM supports TLSv1.3 [2024-10-21T21:02:16,833][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-1] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively [2024-10-21T21:02:16,844][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-1] uncaught exception in thread [main] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.17.1.jar:2.17.1] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1] uncaught exception in thread [main] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:515) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:442) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:515) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:442) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1] ... 6 more Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:486) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:300) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:206) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:252) ~[?:?] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) ~[?:?] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:515) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:442) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1] ... 6 more Caused by: org.opensearch.OpenSearchException: Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1137) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:278) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:456) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:300) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:206) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:252) ~[?:?] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) ~[?:?] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.plugins.PluginsService.(PluginsService.java:197) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:515) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.node.Node.(Node.java:442) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1] ... 6 more java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath] at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1137) at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:278) at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:456) at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:300) at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:206) at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:252) at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:318) at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) at org.opensearch.plugins.PluginsService.(PluginsService.java:197) at org.opensearch.node.Node.(Node.java:515) at org.opensearch.node.Node.(Node.java:442) at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

To Reproduce Steps to reproduce the behavior:

1. Download the values.yaml from https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch
2. Modify file to add initial password
3. Install the helm chart on a gke cluster
4. See exception above

Expected behavior An operational opensearch cluster. Can you provide a values.yaml that can simply be used for development purposes

Chart Name opensearch

Screenshots

Host/Environment (please complete the following information):

• Helm Version: 3.1.14
• Kubernetes Version: 1.30.5-gke.1014001 | 3
• OpenSearch version:2.17.1

Additional context The documentation on open search's website is very outdated. I am following the instructions provided on the root README.md and charts/opensearch/README.md.

I noticed an issue reported on https://github.com/opensearch-project/helm-charts/issues/587. I attempted the same approach and it didn't work.

[Divyaasm]

Hi @pedrocassalpacheco , have you tried using this approach!

[DandyDeveloper]

@pedrocassalpacheco I'm a little confused by this problem.

If you're copying the original issue, you're disabling the Demo config

  - name: DISABLE_INSTALL_DEMO_CONFIG
     value: "false"

This will in turns will NOT provision the TLS certificates that are required (mandatory) for the transport layer between the nodes. Opensearch simply will refuse to start even when you forcefully disable this on the transport level.

The error you seem to be getting, either implies you're mounting your own certificates - In which case you will need to check the securityGroup / fsGRoup are being set appropriately, OR there's something specific in your CRI that is messing with the filesystem / user perms of the files being generated by the demo installation.

I've tried a few different approaches and with the following:

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "false"
  - name: plugins.security.ssl.http.enabled
    value: "false"
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: PasswoCheck@123

My cluster runs fine.

I have problems when I disable the DEMO_CONFIG, which is expected, because certificates are no longer provisioned. The error is also different though:

Likely root cause: OpenSearchException[Wrong Transport SSL configuration. One of Keystore and Truststore files or X.509 PEM certificates and PKCS#8 keys groups should be set to configure Transport layer properly]

Which makes sense because the files don't exist.