OpenSearch
OpenSearch copied to clipboard
opensearch-project
[BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT intermittent failure
Describe the bug
> Task :plugins:repository-s3:yamlRestTest
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Register a repository with a non existing client}" -Dtests.seed=25A4942AE204EF9D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-BH -Dtests.timezone=America/La_Paz -Druntime.java=17 -Dtests.rest.blacklist=repository_s3/50_repository_ecs_credentials/*
org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Register a repository with a non existing client} FAILED
java.lang.RuntimeException: Failure at [repository_s3/20_repository_permanent_credentials:7]: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
at __randomizedtesting.SeedInfo.seed([25A4942AE204EF9D:ADF0ABF04CF88265]:0)
at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:444)
at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.test(OpenSearchClientYamlSuiteTestCase.java:406)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by:
java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:905)
at org.opensearch.client.RestClient.performRequest(RestClient.java:307)
at org.opensearch.client.RestClient.performRequest(RestClient.java:310)
at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
at org.opensearch.test.rest.yaml.ClientYamlTestClient.callApi(ClientYamlTestClient.java:212)
at org.opensearch.test.rest.yaml.ClientYamlTestExecutionContext.callApiInternal(ClientYamlTestExecutionContext.java:263)
at org.opensearch.test.rest.yaml.ClientYamlTestExecutionContext.callApi(ClientYamlTestExecutionContext.java:130)
at org.opensearch.test.rest.yaml.section.DoSection.execute(DoSection.java:306)
at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:431)
... 40 more
Caused by:
java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 more
Coming from https://github.com/opensearch-project/OpenSearch/pull/1917#issuecomment-1028468798
https://github.com/opensearch-project/OpenSearch/pull/3258#pullrequestreview-966650088
Another failure here: https://build.ci.opensearch.org/job/gradle-check/977/
"root_cause" : [
{
"type" : "snapshot_exception",
"reason" : "[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository",
"stack_trace" : "SnapshotException[[repository_permanent:snapshot-one/B67YkZ99R9uEZ5BpNEeGJA] failed to update snapshot in repository]; nested: IllegalStateException[Duplicate key docs (attempted merging values [docs/wOU9hO-vSwOVcRDbp9c4DA] and [docs/Anw_fnI_R56Tjjx1Lb9fOQ])];
at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$37(BlobStoreRepository.java:1332)
at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88)
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:82)
at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
at org.opensearch.action.ActionRunnable$1.doRun(ActionRunnable.java:61)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:798)
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: Duplicate key docs (attempted merging values [docs/wOU9hO-vSwOVcRDbp9c4DA] and [docs/Anw_fnI_R56Tjjx1Lb9fOQ])
at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opensearch.repositories.RepositoryData.<init>(RepositoryData.java:144)
at org.opensearch.repositories.RepositoryData.addSnapshot(RepositoryData.java:371)
at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$39(BlobStoreRepository.java:1354)
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)
... 7 more
Other failure with different error message. 1 https://github.com/opensearch-project/OpenSearch/pull/4115#issuecomment-1204413850 https://build.ci.opensearch.org/job/gradle-check/1383/
> Task :plugins:repository-s3:yamlRestTest
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Delete a non existing snapshot}" -Dtests.seed=6AEAC5574EE779EE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=pt-BR -Dtests.timezone=Africa/Nairobi -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*
org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Delete a non existing snapshot} FAILED
java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:22]: expected [2xx] status code but api [snapshot.delete] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_exception","reason":"[repository_permanent] concurrent modification of the index-N file, expected current generation [11716] but it was not found in the repository","stack_trace":"RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [11716] but it was not found in the repository]
org.opensearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2199)
org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$57(BlobStoreRepository.java:2080)
...
2 https://github.com/opensearch-project/OpenSearch/pull/4122#issuecomment-1204674532 https://build.ci.opensearch.org/job/gradle-check/1416/
> Task :plugins:repository-s3:yamlRestTest
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=D2A4C945B309FD46 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-LB -Dtests.timezone=Africa/Dakar -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*
org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} FAILED
java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:244]: expected [2xx] status code but api [snapshot.restore] returned [404 Not Found] [{"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[repository_permanent:snapshot-one/8mXh1LBqTjSq-uZOhcgV4Q] is missing","stack_trace":"SnapshotMissingException[[repository_permanent:snapshot-one/8mXh1LBqTjSq-uZOhcgV4Q] is missing]; nested: NoSuchFileException[Blob object [****_integration_tests/snap-8mXh1LBqTjSq-uZOhcgV4Q.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: HAZNQYZ4400KNKKC; S3 Extended Request ID: Lghc26ALGZhmkwujL6PLA/xP9N9eoizpyqOOJodxe/xpnruMtTJomktTx+QxrjmbX7nF/iNghUM=; Proxy: null)];
org.opensearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:1505)\n\tat org.opensearch.snapshots.RestoreService.lambda$restoreSnapshot$2(RestoreService.java:231)\n\tat
...
3 https://github.com/opensearch-project/OpenSearch/pull/4123#issuecomment-1204670140 https://build.ci.opensearch.org/job/gradle-check/1417/
> Task :plugins:repository-s3:yamlRestTest
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=A25F370B6CEC23C4 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-MX -Dtests.timezone=America/Iqaluit -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*
org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} FAILED
java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:152]: expected [2xx] status code but api [snapshot.create] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists","stack_trace":"InvalidSnapshotNameException[[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists]
org.opensearch.snapshots.SnapshotsService.ensureSnapshotNameAvailableInRepo(SnapshotsService.java:689)org.opensearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:428)
...
Looks like I am facing similar issue when using open-search-high-level-client java client from android. Lot of retries happens, works some times. From Another standalone java seems working fine. Is there any workaround for this one.
2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: 12:00:06.637 [IntentService[ServiceIntent]] ERROR - openearch read: java.io.IOException: Unable to decrypt incoming data due to unexpected end of stream 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:912) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestClient.performRequest(RestClient.java:301) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestClient.performRequest(RestClient.java:289) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at org.opensearch.client.RestHighLevelClient.searchTemplate(RestHighLevelClient.java:1306) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.opensearch.OpenSearchAccess.searchTemplateWithRetry(OpenSearchAccess.java:404) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.opensearch.OpenSearchAccess.getLatestMessageByTimeStampAndEpoch(OpenSearchAccess.java:598) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.opensearch.AtomicData.getSearchHits(AtomicData.java:117) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.opensearch.AtomicData.queryForData(AtomicData.java:156) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.opensearch.BdsAtomicData.getSateliteData(BdsAtomicData.java:61) 2022-09-13 12:00:06.681 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.rtcm.RtcmMsgSet.loadRtcmMsgCacheFromDb(RtcmMsgSet.java:753) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.am.AMManager.loadRtcmMsgCacheFromDb(AMManager.java:441) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.pm.WeightedLeastSquares.updateAssistReplayMode(WeightedLeastSquares.java:3844) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.pm.WeightedLeastSquares.calculateUserPositionVelocityLeastSquare(WeightedLeastSquares.java:283) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.pm.PMController.computePositionVelocitySolutionsFromRawMeas(PMController.java:644) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at ai.onenav.pm.PE_Fix.handleStrobeToFix(PE_Fix.java:1696) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at io.onenav.ceshostapp.Rinse.processMeasForNewFixAndPrePos(Rinse.java:5253) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at io.onenav.ceshostapp.MainLoop.mainLoop(MainLoop.java:713) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at io.onenav.ceshostapp.ServiceIntent.onHandleIntent(ServiceIntent.java:345) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:78) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at android.os.Handler.dispatchMessage(Handler.java:107) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at android.os.Looper.loop(Looper.java:214) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at android.os.HandlerThread.run(HandlerThread.java:67) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: Caused by: javax.net.ssl.SSLException: Unable to decrypt incoming data due to unexpected end of stream 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.nio.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:497) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:526) 2022-09-13 12:00:06.682 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) 2022-09-13 12:00:06.684 9068-9334/io.onenav.ceshostapp I/System.out: at java.lang.Thread.run(Thread.java:919)
Another failure, https://build.ci.opensearch.org/job/gradle-check/5123/ impacting https://github.com/opensearch-project/OpenSearch/pull/4843
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=5A6D28597C359FE7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-DO -Dtests.timezone=Atlantic/Canary -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*
I am going to close this issue and open individual ones for separate RepositoryS3ClientYamlTestSuiteIT failures.
https://github.com/opensearch-project/OpenSearch/issues/5219