QA - Elasticsearch Client internal error exception with Datastore logout in Cucumber tests

[gbarbon]

Describe the bug The following exception org.eclipse.kapua.service.elasticsearch.client.exception.ClientInternalError: Internal client error! Error: Error in handling REST timeout handler is thrown when logging out from the Datastore in Cucumber tests. This does not cause test failures, thus following tests are executed correctly, however this exception should not appear. For further information, see the stacktrace attached below.

To Reproduce Steps to reproduce the behavior:

1. Launch the following tests with maven: mvn -B -Dcommons.db.schema=kapuadb -Dcommons.settings.hotswap=true -Dbroker.host=localhost -Dgroups='!org.eclipse.kapua.qa.markers.junit.JUnitTests' -Dcucumber.options="--tags @user" verify
2. See error

This does not happen regularly. I first noticed it while trying in GitHub Actions the 'User' Cucumber tests. I then noticed this when launching the same tests locally on my machine. However, the failure did not arise in the same point. In the first case it arose in the cleanup at the beginning of Cucumber feature User Permission tests, before the first scenario, while in the second case it arose in the cleanup at the end of the last scenario of the same Cucumber feature and in the cleanup at the beginning of the following feature test.

Expected behavior No exceptions should be obtained when logging out from the Datastore.

Screenshots n/a

Version of Kapua 1.4.0-SNAPSHOT

Type of deployment [ ] Local Vagrant deployment [ ] Docker [ ] Openshift (in its variants) [x] Others : Cucumber QA tests with maven

Main component affected [ ] Console (in case of console please report info on which browser you encountered the problem) [ ] REST API [ ] Message Broker [x] - Others

Additional context The stacktrace with the exception:

12:16:15.208 [main] INFO  o.e.k.s.d.steps.DatastoreSteps - Logging out in cleanup
12:16:15.218 [main] ERROR o.e.k.s.d.steps.DatastoreSteps - Failed to log out in @After
org.eclipse.kapua.service.elasticsearch.client.exception.ClientInternalError: Internal client error! Error: Error in handling REST timeout handler
	at org.eclipse.kapua.service.elasticsearch.client.rest.RestElasticsearchClient.restCallTimeoutHandler(RestElasticsearchClient.java:603)
	at org.eclipse.kapua.service.elasticsearch.client.rest.RestElasticsearchClient.deleteAllIndexes(RestElasticsearchClient.java:534)
	at org.eclipse.kapua.service.datastore.internal.MessageStoreFacade.deleteAllIndexes(MessageStoreFacade.java:530)
	at org.eclipse.kapua.service.datastore.internal.mediator.DatastoreMediator.deleteAllIndexes(DatastoreMediator.java:238)
	at org.eclipse.kapua.service.datastore.steps.DatastoreSteps.deleteIndices(DatastoreSteps.java:366)
	at org.eclipse.kapua.service.datastore.steps.DatastoreSteps.afterScenario(DatastoreSteps.java:349)
	at sun.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at cucumber.runtime.Utils$1.call(Utils.java:37)
	at cucumber.runtime.Timeout.timeout(Timeout.java:13)
	at cucumber.runtime.Utils.invoke(Utils.java:31)
	at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
	at cucumber.runtime.Runtime.runHookIfTagsMatch(Runtime.java:223)
	at cucumber.runtime.Runtime.runHooks(Runtime.java:211)
	at cucumber.runtime.Runtime.runAfterHooks(Runtime.java:205)
	at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:46)
	at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:91)
	at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:63)
	at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:18)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:70)
	at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)
	at cucumber.api.junit.Cucumber.runChild(Cucumber.java:37)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at cucumber.api.junit.Cucumber.run(Cucumber.java:98)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
Caused by: java.net.ConnectException: Connection refused
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
	at org.eclipse.kapua.service.elasticsearch.client.rest.RestElasticsearchClient.lambda$deleteAllIndexes$13(RestElasticsearchClient.java:536)
	at org.eclipse.kapua.service.elasticsearch.client.rest.RestElasticsearchClient.restCallTimeoutHandler(RestElasticsearchClient.java:582)
	... 52 common frames omitted
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.lang.Thread.run(Thread.java:748)