[bitnami/elasticsearch] `plugin.mandatory` misnaming for external plugins causes startup to fail.

[peterdm]

trafficstars

Description

External elasticsearch plugins install successfully, but are misnamed when copied to plugin.mandatory causing each node to fail to start with an exception.

Steps to reproduce the issue:

docker run --name test -e ELASTICSEARCH_PLUGINS=http://es-learn-to-rank.labs.o19s.com/ltr-plugin-v1.5.1-es7.9.2.zip -d bitnami/elasticsearch:7.9.2-debian-10-r0

Describe the results you received:

Fails to start with exception:

...
uncaught exception in thread [main]
java.lang.IllegalStateException: missing mandatory plugins [ltr-plugin-v1.5.1-es7.9.2], found plugins [ltr, repository-s3]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:186)

Describe the results you expected:

Bitnami populates the plugin.mandatory value from ELASTICSEARCH_PLUGINS which prevents a search node from starting if all plugins are not successfully installed. This works well with default (included) plugins which Elasticsearch will install by "plugin name". However when using an external plugin resource (zipfile over http or local filesystem), even though the plugin is installed correctly, the plugin name used to populate plugin.mandatory is not correctly derived from the plugin resource.

In the example to reproduce (above), the plugin (named ltr) is correctly installed, however, the value copied to plugin.mandatory is ltr-plugin-v1.5.1-es7.9.2 which fails the startup check, since it is not also ltr.

Additional information you deem important (e.g. issue happens only occasionally):

There is an attempt to derive the "plugin name" from the downloaded filename here: https://github.com/bitnami/bitnami-docker-elasticsearch/blob/master/7/debian-10/rootfs/opt/bitnami/scripts/libelasticsearch.sh#L500

However there is no such plugin file-naming convention required by Elastic. Instead, there is a plugin-descriptor.properties inside every external elasticsearch plugin archive containing a required property name which should be used instead.

Version

• Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 19
  Running: 0
  Paused: 0
  Stopped: 19
 Images: 32
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 2.924GiB
 Name: docker-desktop
 ID: CWGO:3SKD:BICU:JSQL:7RFU:MVGE:UPLF:EQVJ:VJSP:GCBO:NTHO:FQQY
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 35
  Goroutines: 52
  System Time: 2020-10-23T17:13:45.597921481Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

• Output of docker-compose version (if applicable):

(paste your output here)

Additional environment details (AWS, VirtualBox, Docker for MAC, physical, etc.):

[miguelaeh]

Hi @peterdm , Thank you very much for reporting this. Would you be able to create a PR adding what you explained in the additional informatin?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[miguelaeh]

Hi @peterdm , Any news on this?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[phoenixgao]

Any news?

I have the same problem when installing external plugins

[miguelaeh]

Hi @phoenixgao , It seems there is no news on this. I will create an internal task to check it.

[HEYGUL]

I confirm the issue still exists. Any news on it ? BTW, if you have any workaround, it would be great, as I have no clue on how to ?

[miguelaeh]

Hi @HEYGUL , By the moment the only workaround is to download the plugins with the same name they will be added to the mandatory list.

[terryzwt]

Any progress on this? I face the same problem when install https://github.com/medcl/elasticsearch-analysis-ik/ test command:

docker run --name test -e ELASTICSEARCH_PLUGINS=https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v8.2.0/elasticsearch-analysis-ik-8.2.0.zip -d docker.io/bitnami/elasticsearch:8.2.0-debian-10-r1

docker logs -f test output:

[2022-05-17T18:06:42,780][INFO ][o.e.n.Node               ] [8be7d05f63b7] version[8.2.0], pid[1], build[default/tar/b174af62e8dd9f4ac4d25875e9381ffe2b9282c5/2022-04-20T10:35:10.180408517Z], OS[Linux/3.10.0-1062.18.1.el7.x86_64/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7-LTS]
[2022-05-17T18:06:42,782][INFO ][o.e.n.Node               ] [8be7d05f63b7] JVM home [/opt/bitnami/java], using bundled JDK [false]
[2022-05-17T18:06:42,782][INFO ][o.e.n.Node               ] [8be7d05f63b7] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11468262786408555469, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
uncaught exception in thread [main]
[2022-05-17T18:06:45,808][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [8be7d05f63b7] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: missing mandatory plugins [elasticsearch-analysis-ik], found plugins [analysis-ik]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-8.2.0.jar:8.2.0]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-8.2.0.jar:8.2.0]
Caused by: java.lang.IllegalStateException: missing mandatory plugins [elasticsearch-analysis-ik], found plugins [analysis-ik]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:187) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:393) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:291) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.2.0.jar:8.2.0]
	... 6 more
java.lang.IllegalStateException: missing mandatory plugins [elasticsearch-analysis-ik], found plugins [analysis-ik]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:187)
	at org.elasticsearch.node.Node.<init>(Node.java:393)
	at org.elasticsearch.node.Node.<init>(Node.java:291)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /opt/bitnami/elasticsearch/logs/elasticsearch.log

[terryzwt]

Hi @HEYGUL , By the moment the only workaround is to download the plugins with the same name they will be added to the mandatory list.

when install the elasticsearch via helm using bitnami charts, it's impossible to handle it.

[fmulero]

We are going to transfer this issue to bitnami/containers

In order to unify the approaches followed in Bitnami containers and Bitnami charts, we are moving some issues in bitnami/bitnami-docker-<container> repositories to bitnami/containers.

Please follow bitnami/containers to keep you updated about the latest bitnami images.

More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.