[CI] SearchWithMinCompatibleSearchNodeIT testMinVersionAsNewVersion failing

[elasticsearchmachine]

trafficstars

Build Scans:

• elasticsearch-periodic #4658 / 7.2.1_bwc
• elasticsearch-periodic #4607 / 7.5.2_bwc

Reproduction Line:

./gradlew ':qa:mixed-cluster:v7.2.1#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT" -Dtests.method="testMinVersionAsOldVersion" -Dtests.seed=C9112497E5F5BC78 -Dtests.bwc=true -Dtests.locale=ar-KW -Dtests.timezone=Africa/Asmara -Druntime.java=22

Applicable branches: 7.17

Reproduces locally?: N/A

Failure History: See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:34195], URI [test_min_version/_doc/6], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [test_min_version/_doc/6]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [test_min_version/_doc/6]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

Issue Reasons:

• [7.17] 2 failures in test testMinVersionAsNewVersion (1.0% fail rate in 191 executions)
• [7.17] 2 failures in pipeline elasticsearch-periodic (18.2% fail rate in 11 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

I think the root cause is the following:

» [2024-08-23T12:29:52,372][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.3.2-3] fatal error in thread [elasticsearch[v7.3.2-3][clusterApplierService#updateTask][T#1]], exiting
»  java.lang.AssertionError: {_doc=org.elasticsearch.index.mapper.DocumentMapper@594475ec}
»  	at org.elasticsearch.index.mapper.MapperService.assertMappingVersion(MapperService.java:272) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:226) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.index.IndexService.updateMapping(IndexService.java:581) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.updateIndices(IndicesClusterStateService.java:538) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:263) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:495) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:464) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:418) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.3.2.jar:7.3.2]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
»  	at java.lang.Thread.run(Thread.java:835) [?:?]

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

Labeling as "low risk" since from looking at the test failure history, this only occasionally fails on the 7.17 branch in mixed cluster bwc tests and thus looks quite limited in scope.

[drempapis]

This class has been unmuted #115386 for v7.17 and permanently deleted for v9.0 #114713 I close this issue