elasticsearch
elasticsearch copied to clipboard
elastic
Add mixed mode BWC testing support for test-specific plugins
This issues been filed to examine how best to support the inference-service-test plugin in ES|QL mixed version testing.
The ES|QL CSV and REST tests run with a variety of modes (see x-pack/plugin/esql/qa/server):
single-nodemulti-nodemixed-clustermulti-clusters
No.1 and no.2 run the same ES version on all nodes. No.3 and no.4 run with mixed ES versions, either within the same cluster or across clusters.
The issue occurs only with no.3 and no.4 where the BWC infrastructure is not able to install the interference service test-specific plugin into the older nodes. When we try it, we get the following error:
> Task :x-pack:plugin:esql:qa:server:mixed-cluster:v8.17.0#javaRestTest FAILED
MixedClusterEsqlSpecIT > classMethod FAILED
java.lang.RuntimeException: An error occurred orchestrating test cluster.
at __randomizedtesting.SeedInfo.seed([27B9DA09AA60FCA0]:0)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:270)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:275)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:78)
at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:46)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by:
java.lang.IllegalArgumentException: Can't run executable: `/Users/chegar/git/elasticsearch-lucene_snapshot/x-pack/plugin/esql/qa/server/mixed-cluster/build/testrun/v8.17.0_javaRestTest/temp/test-cluster8788876539987453686/test-cluster-2/distro/bin/elasticsearch-plugin` does not exist.
at org.elasticsearch.test.cluster.util.ProcessUtils.exec(ProcessUtils.java:54)
at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.runToolScript(AbstractLocalClusterFactory.java:784)
at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.installPlugins(AbstractLocalClusterFactory.java:633)
at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:159)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:78)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:785)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:78)
at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:276)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1491)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2073)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2035)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
@ChrisHegarty do you have a branch/open PR I can pull to repro the error locally?
@ChrisHegarty do you have a branch/open PR I can pull to repro the error locally?
This commit enables multi-versioning testing in ES|QL, so reproduces the issue: https://github.com/ChrisHegarty/elasticsearch/commit/82a638183e9ae737ea04744f9203a95862e0bb50
Then just run, e.g. ./gradlew :x-pack:plugin:esql:qa:server:mixed-cluster:v8.17.0#javaRestTest, to get output
...
MixedClusterEsqlSpecIT > classMethod FAILED
java.lang.RuntimeException: An error occurred orchestrating test cluster.
at __randomizedtesting.SeedInfo.seed([CF3C42D5E03B6ED4]:0)
...
Run with ./gradlew ... --info to see the logs, e.g.
...
[2024-10-25T04:20:10,409][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-0]] Installing plugins [inference-service-test] into node 'test-cluster-2
[2024-10-25T04:20:10,410][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-2]] Installing plugins [inference-service-test] into node 'test-cluster-0
[2024-10-25T04:20:10,452][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-1]] Installing plugins [inference-service-test] into node 'test-cluster-1
[2024-10-25T04:20:10,452][INFO ][o.e.t.c.l.AbstractLocalClusterFactory] [[test-cluster-node-executor-3]] Installing plugins [inference-service-test] into node 'test-cluster-3
[2024-10-25T04:20:11,790][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:11,790][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:11,848][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:11,849][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:13,828][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installed inference-service-test
[2024-10-25T04:20:13,828][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Please restart Elasticsearch to activate any plugins installed
[2024-10-25T04:20:13,841][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installed inference-service-test
[2024-10-25T04:20:13,841][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Please restart Elasticsearch to activate any plugins installed
[2024-10-25T04:20:15,427][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:15,427][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:15,428][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:15,428][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:15,443][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Failed installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
[2024-10-25T04:20:15,443][INFO ][process-output ] [[elasticsearch-plugin-log-forwarder]] -> Failed installing file:///Users/chegar/git/tmp/elasticsearch/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-9.0.0-SNAPSHOT.zip
...
We are hitting this issue as well with semantic text tests. We want to create a mixed-version cluster for BwC testing and install the inference-service-test plugin on it. However, the plugin install fails because the plugin is built only for the current ES version.
The issue with java.lang.IllegalArgumentException: Can't run executable: [...]/distro/bin/elasticsearch-plugin does not exist. is possibly related to the test cluster setup bug that has been fixed in https://github.com/elastic/elasticsearch/pull/124628 .
@jozala - tested to see if the problem is now resolved.
It's not - but we do get a different error this time:
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] java.lang.IllegalArgumentException: Plugin [inference-service-test] was built for Elasticsearch version 9.1.0 but version 9.0.0 is running
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] at org.elasticsearch.plugins.PluginsUtils.verifyCompatibility(PluginsUtils.java:148)
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] at org.elasticsearch.plugins.cli.InstallPluginAction.loadPluginInfo(InstallPluginAction.java:867)
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] at org.elasticsearch.plugins.cli.InstallPluginAction.installPlugin(InstallPluginAction.java:923)
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] at org.elasticsearch.plugins.cli.InstallPluginAction.execute(InstallPluginAction.java:254)
[2025-04-03T11:42:30,915][ERROR][process-output ][[elasticsearch-plugin-log-forwarder]] at org.elasticsearch.plugins.cli.InstallPluginCommand.execute(InstallPluginCommand.java:90)
@ioanatia I think this new error is unrelated to the test cluster setup bug I've mentioned above. I suspect the test cluster setup bug could hide this one before. I'm not sure where you're running those tests, but if they're BwC run on the branch, then updating the branch with the changes from main often helps with those versions mixups.