elasticsearch icon indicating copy to clipboard operation
elasticsearch copied to clipboard

Published 20 hours ago • verified publisher icon elastic

[CI] DeprecationHttpIT testDeprecatedSettingsReturnWarnings failing

Open mosche opened this issue 9 months ago • 4 comments

Build scan: https://gradle-enterprise.elastic.co/s/nlw52dnqiof4a/tests/:x-pack:plugin:deprecation:qa:rest:javaRestTest/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings

Reproduction line:

./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=94B983B3041BD3C8 -Dtests.locale=ga -Dtests.timezone=Africa/Algiers -Druntime.java=21

Applicable branches: main

Reproduces locally?: No

Failure history: Failure dashboard for org.elasticsearch.xpack.deprecation.DeprecationHttpIT#testDeprecatedSettingsReturnWarnings

Failure excerpt:

java.lang.AssertionError: 
Expected: a collection with size <2>
     but: collection size was <0>

  at __randomizedtesting.SeedInfo.seed([94B983B3041BD3C8:B992366E42E13572]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2188)
  at org.elasticsearch.xpack.deprecation.DeprecationHttpIT.lambda$testDeprecatedSettingsReturnWarnings$1(DeprecationHttpIT.java:142)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1294)
  at org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings(DeprecationHttpIT.java:139)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

mosche avatar May 14 '24 15:05 mosche

Pinging @elastic/es-core-infra (Team:Core/Infra)

elasticsearchmachine avatar May 14 '24 15:05 elasticsearchmachine

I briefly peeked into a recent failure (buildscan), the test timed out waiting 30secs for the deprecation to appear in logs. To me doesn't look like a test failure, but I'd have to look further into it.

The test logs:

[2024-05-07T10:01:00,440][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] before test
мај 07, 2024 10:01:00 AM org.elasticsearch.client.RestClient logResponse
WARNING: request [PUT http://127.0.0.1:40969/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]
[2024-05-07T10:01:00,963][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:00,989][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,031][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,087][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,122][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,192][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,265][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,370][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,535][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,830][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:02,379][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:03,451][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:05,544][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:09,674][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:17,906][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:34,318][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
мај 07, 2024 10:01:34 AM org.elasticsearch.client.RestClient logResponse
WARNING: request [PUT http://[::1]:41399/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]
[2024-05-07T10:01:34,466][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] after test
REPRODUCE WITH: ./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=446C182627B25D32 -Dtests.locale=sr-ME -Dtests.timezone=US/Mountain -Druntime.java=21

However, looking at the deprecation log you'll see the following:

{"@timestamp":"2024-05-07T16:01:34.325Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"dDcqp0Y3Tte6TEqtUezFsw","elasticsearch.node.id":"3E_sV7YrQJW0ySfuirx-0g","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-07T16:01:34.326Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"dDcqp0Y3Tte6TEqtUezFsw","elasticsearch.node.id":"3E_sV7YrQJW0ySfuirx-0g","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}

These logs correspond to the request in the finally block, when resetting/removing the deprecated settings again. It appears as if the first request was neither logged nor indexed 🤷

mosche avatar May 14 '24 15:05 mosche

Another at https://gradle-enterprise.elastic.co/s/nlw52dnqiof4a/tests/task/:x-pack:plugin:deprecation:qa:rest:javaRestTest/details/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings?top-execution=1

mosche avatar May 14 '24 15:05 mosche

test logs

[2024-05-13T16:43:56,601][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] before test	
Beal 13, 2024 4:43:56 P.M. org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://[::1]:44625/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2024-05-13T16:43:56,903][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,921][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,944][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,965][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,994][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,033][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,086][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,173][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,329][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,610][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:58,143][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:59,190][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:01,259][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:05,375][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:13,588][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:30,000][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
Beal 13, 2024 4:44:30 P.M. org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://127.0.0.1:43361/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2024-05-13T16:44:30,105][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] after test	
REPRODUCE WITH: ./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=94B983B3041BD3C8 -Dtests.locale=ga -Dtests.timezone=Africa/Algiers -Druntime.java=21

deprecation logs

{"@timestamp":"2024-05-13T15:43:56.712Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecationIndexingCacheReset","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:43:56.718Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecationIndexingCacheReset","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:44:30.010Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:44:30.011Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}

mosche avatar May 14 '24 15:05 mosche

Happened again - https://gradle-enterprise.elastic.co/s/q7tq33jubmjxk/tests/:x-pack:plugin:deprecation:qa:rest:javaRestTest/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings

thecoop avatar May 23 '24 13:05 thecoop

The deprecation log of above test failure suggests a bug in the code: apparently we are occasionally loosing the x-opaque-id on the way :(

{"@timestamp":"2024-05-23T13:21:27.784Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.event.category":"settings","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"whdPc5jFRhmK165USgd4yQ","elasticsearch.node.id":"tLChz9h8Rx2RduqH2zXDAQ","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-23T13:21:27.805Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.event.category":"settings","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"whdPc5jFRhmK165USgd4yQ","elasticsearch.node.id":"tLChz9h8Rx2RduqH2zXDAQ","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-23T13:22:02.146Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings-cleanup","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"whdPc5jFRhmK165USgd4yQ","elasticsearch.node.id":"tLChz9h8Rx2RduqH2zXDAQ","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-23T13:22:02.147Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings-cleanup","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"whdPc5jFRhmK165USgd4yQ","elasticsearch.node.id":"tLChz9h8Rx2RduqH2zXDAQ","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}

Matching test logs are:

[testDeprecatedSettingsReturnWarnings] before test	
[2567-05-23T15:21:26,497][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] initializing REST clients against [http://[::1]:57150, http://127.0.0.1:57151/]	
[2567-05-23T15:21:27,204][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] This test is running on the legacy test framework; historical features from production code will not be available. You need to port the test to the new test plugins in order to use historical features from production code. If this is a legacy feature used only in tests, you can add it to a test-only FeatureSpecification such as org.elasticsearch.test.rest.RestTestLegacyFeatures.	
?.?. 23, 2024 3:21:27 ?????????? org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://127.0.0.1:57151/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2567-05-23T15:21:29,223][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:29,493][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:29,767][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:30,298][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:31,344][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:33,409][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:37,527][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:21:45,738][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2567-05-23T15:22:02,140][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
?.?. 23, 2024 3:22:02 ?????????? org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://[::1]:57150/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2567-05-23T15:22:02,210][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] Warning headers on cleanup: [299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release.", 299 Elasticsearch-8.15.0-8bca164cbdbdad78ad2a2a16a8bf7ba191a276a1 "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2567-05-23T15:22:02,237][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] after test	
REPRODUCE WITH: gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=26875D96E39FE9DE -Dtests.locale=th-TH -Dtests.timezone=Europe/Vienna -Druntime.java=22

mosche avatar May 23 '24 15:05 mosche

Another failure: https://gradle-enterprise.elastic.co/s/i6xilq3ee5unk/tests/:x-pack:plugin:deprecation:qa:rest:javaRestTest/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings

nielsbauman avatar May 24 '24 08:05 nielsbauman

Another failure: https://gradle-enterprise.elastic.co/s/e3excygnaxr6e

jfreden avatar May 30 '24 07:05 jfreden

Another failure https://gradle-enterprise.elastic.co/s/thbd6du5leywq

maxhniebergall avatar Jun 03 '24 14:06 maxhniebergall

I can see that the ThreadContext (headers) are getting list between submitting the update task to the master task queue and eventually executing it. Though it's not happening all the time, I'll continue investigating later to figure out the difference.

It looks like the behavior has changed here as well. I remember that the test was previously barley failing, now it's the opposite. It hardly ever succeeds.

mosche avatar Aug 07 '24 10:08 mosche