elasticsearch [CI] SearchWithMinCompatibleSearchNodeIT testCcsMinimizeRoundtripsIsFalse failing

Other 3 tests in the same suite failed in this run for the same reason: Index 0 out of bounds for length 0

On a first look it seems similar to https://github.com/elastic/elasticsearch/issues/83903, but this test is on the 7.x branch, so it might be unrelated.

Build scan: https://gradle-enterprise.elastic.co/s/z7wqvjof4u3n6/tests/:qa:mixed-cluster:v7.1.1%23mixedClusterTest/org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT/testCcsMinimizeRoundtripsIsFalse Reproduction line:

./gradlew ':qa:mixed-cluster:v7.1.1#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT" -Dtests.method="testCcsMinimizeRoundtripsIsFalse" -Dtests.seed=306C0CB8C36996E2 -Dtests.bwc=true -Dtests.locale=id -Dtests.timezone=Africa/Juba -Druntime.java=21

Applicable branches: 7.17

Reproduces locally?: Didn't try

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT&tests.test=testCcsMinimizeRoundtripsIsFalse

Failure excerpt:

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

  at __randomizedtesting.SeedInfo.seed([306C0CB8C36996E2:31ED264F8DC4E415]:0)
  at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
  at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
  at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
  at java.util.Objects.checkIndex(Objects.java:385)
  at java.util.ArrayList.get(ArrayList.java:427)
  at org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT.prepareTestData(SearchWithMinCompatibleSearchNodeIT.java:57)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

Nov 09 '23 15:11 ldematte

Pinging @elastic/es-search (Team:Search)

Nov 09 '23 15:11 elasticsearchmachine

This is a weird test failure.

It fails on the line:

    newVersion = newNodes.get(0).getVersion();

It isn't failing due to any production bug, seems like a bug in the testing infra.

Nov 09 '23 18:11 benwtrent

This is failing nearly every day in our 7.17 BWC jobs. We might want to consider muting this but not sure we want to lose that coverage as this issue effects multiple tests in this suite.

Nov 19 '23 18:11 mark-vieira

This has been muted on branch main

Mute Reasons:

[main] 3 failures in test testCcsMinimizeRoundtripsIsFalse (0.4% fail rate)
[main] 3 failures in step 7.17.23_bwc-snapshots (0.6% fail rate)
[main] 3 failures in pipeline elasticsearch-pull-request (0.7% fail rate)

Build Scans:

Jun 28 '24 18:06 elasticsearchmachine

Pinging @elastic/es-search-foundations (Team:Search Foundations)

Jul 17 '24 18:07 elasticsearchmachine