vertx-zookeeper icon indicating copy to clipboard operation
vertx-zookeeper copied to clipboard

Published 20 hours ago • verified publisher icon vert-x3

Vertx zookeeper error

Open mscofy opened this issue 6 years ago • 10 comments

I am using vertx with zookeeper as my cluster manager. Suddenly I started getting this excpetion in my logs:

ERROR io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager - Failed to handle memberRemoved io.vertx.core.VertxException: java.io.InvalidClassException: io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap$KeyValue; local class incompatible: stream classdesc serialVersionUID = -472487054773491211, local class serialVersionUID = -6066652933022630380 at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.get(ZKSyncMap.java:95) How can I fix it?

Regards, Ido

mscofy avatar Nov 08 '17 03:11 mscofy

Hi Do you upgrade vert.x from old version to the latest? which version of vert.x your running.

Thanks.

stream-iori avatar Nov 09 '17 05:11 stream-iori

Yes,i upgrade vext.x from 3.4.2 to 3.5.0. Now i'm running version 3.5.0

mscofy avatar Nov 09 '17 06:11 mscofy

We using Java serialization to save object into the zk, this issue would be happen if there are multi version vertx-zookeeper in one cluster, you have to restart all the verticle with latest version of vertx-zookeeper. I hope could refactor this method in the next release.

stream-iori avatar Nov 09 '17 10:11 stream-iori

I also encountered this problem. And I also used a different version of vertx-zookeeper.

[ Curator-PathChildrenCache-0 ] - [ ERROR ] [ i.v.spi.cluster.zookeeper.ZookeeperClusterManager : 167 ] - Failed to handle memberRemoved
io.vertx.core.json.DecodeException: Failed to decode: null
	at io.vertx.core.json.Json.decodeValue(Json.java:124)
	at io.vertx.core.json.JsonObject.fromJson(JsonObject.java:956)
	at io.vertx.core.json.JsonObject.<init>(JsonObject.java:48)
	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:322)
	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)

Now I try to upgrade to the latest version of vertx-zookeeper, and I will give feedback after the results.

aruis avatar Dec 28 '18 01:12 aruis

In all the years we've been running vertx, first with Hazelcast and now with Zookeeper, this failure to connect to a running cluster unless you have the exact same minor version of vertx-hazelcast or vertx-zookeeper as every other verticle in the entire distributed system has been the major pain in the ass and IMHO an odd design choice. We heavily use eventbus message passing between verticles but don't attempt to share state - it might be more effective to use an external messaging service and not cluster the verticles. Of course this is totally doable but seems to defeat what I took as a core capability of Vertx.

ijl20 avatar May 15 '20 10:05 ijl20

I have the same version across the entire cluster, but this issue occurs when I redeploy one of the members. So far the only way to resolve it is to redeploy every node in the cluster. I am currently on version 3.9.1.

Is there anyone looking into this issue. I agree with ijl20. this is a severe problem, and like his system, we are only using clustering for event bus message passing.

This is my exception that i am seeing:

2020-06-20T10:24:16 - SEVERE: Failed to handle memberRemoved
2020-06-20T10:24:16 - io.vertx.core.VertxException: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:156)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.entrySet(ZKSyncMap.java:180)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:320)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
2020-06-20T10:24:16 - 	at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-06-20T10:24:16 - 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-06-20T10:24:16 - Caused by: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:152)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2020-06-20T10:24:16 - 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:154)
2020-06-20T10:24:16 - 	... 20 more
2020-06-20T10:24:16 - Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:310)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:299)
2020-06-20T10:24:16 - 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:296)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:287)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:34)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:149)
2020-06-20T10:24:16 - 	... 28 more
2020-06-20T10:24:22 - Jun 20, 2020 2:24:22 PM io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
2020-06-20T10:24:22 - WARNING: Error removing subs
2020-06-20T10:24:22 - java.io.EOFException
2020-06-20T10:24:22 - 	at java.base/java.io.DataInputStream.readBoolean(DataInputStream.java:249)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKMap.asObject(ZKMap.java:110)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$22(ZKAsyncMultiMap.java:195)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$23(ZKAsyncMultiMap.java:193)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$24(ZKAsyncMultiMap.java:189)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$removeAllMatching$26(ZKAsyncMultiMap.java:187)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.removeAllMatching(ZKAsyncMultiMap.java:186)
2020-06-20T10:24:22 - 	at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.lambda$setClusterViewChangedHandler$12(ClusteredEventBus.java:273)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$checkSubs$12(HAManager.java:520)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$runOnContextAndWait$13(HAManager.java:529)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
2020-06-20T10:24:22 - 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-20T10:24:22 - 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-20T10:24:22 - 	at java.base/java.lang.Thread.run(Thread.java:834)

melaraj2 avatar Jun 20 '20 14:06 melaraj2

I have the same version across the entire cluster, but this issue occurs when I redeploy one of the members. So far the only way to resolve it is to redeploy every node in the cluster. I am currently on version 3.9.1.

Is there anyone looking into this issue. I agree with ijl20. this is a severe problem, and like his system, we are only using clustering for event bus message passing.

This is my exception that i am seeing:

2020-06-20T10:24:16 - SEVERE: Failed to handle memberRemoved
2020-06-20T10:24:16 - io.vertx.core.VertxException: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:156)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.entrySet(ZKSyncMap.java:180)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:320)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
2020-06-20T10:24:16 - 	at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-06-20T10:24:16 - 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-06-20T10:24:16 - Caused by: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:152)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2020-06-20T10:24:16 - 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:154)
2020-06-20T10:24:16 - 	... 20 more
2020-06-20T10:24:16 - Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:310)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:299)
2020-06-20T10:24:16 - 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:296)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:287)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:34)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:149)
2020-06-20T10:24:16 - 	... 28 more
2020-06-20T10:24:22 - Jun 20, 2020 2:24:22 PM io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
2020-06-20T10:24:22 - WARNING: Error removing subs
2020-06-20T10:24:22 - java.io.EOFException
2020-06-20T10:24:22 - 	at java.base/java.io.DataInputStream.readBoolean(DataInputStream.java:249)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKMap.asObject(ZKMap.java:110)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$22(ZKAsyncMultiMap.java:195)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$23(ZKAsyncMultiMap.java:193)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$24(ZKAsyncMultiMap.java:189)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$removeAllMatching$26(ZKAsyncMultiMap.java:187)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.removeAllMatching(ZKAsyncMultiMap.java:186)
2020-06-20T10:24:22 - 	at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.lambda$setClusterViewChangedHandler$12(ClusteredEventBus.java:273)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$checkSubs$12(HAManager.java:520)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$runOnContextAndWait$13(HAManager.java:529)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
2020-06-20T10:24:22 - 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-20T10:24:22 - 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-20T10:24:22 - 	at java.base/java.lang.Thread.run(Thread.java:834)

This exception should have been swallow and make a WARNING to log. I am workng on vertx4-zookeeper, and this exception should convert to warning log

stream-iori avatar Jun 20 '20 16:06 stream-iori

The problem is not just the exception, in my case I have a JVM instance that calls eventBus.request to an address being consumed by a set of JVMs, when I restart the consumers, the requests to that address fail with timeout exceptions. It appears that the only way to resolve it is by also restarting the requester JVMs on other instances. In my case, it is a normal course of business to update the consumers in the cluster. I imagine this is a common thing.

All of this runs on AWS Fargate, so its important to note that IP addresses change when i restart the service.

melaraj2 avatar Jun 21 '20 03:06 melaraj2

the sending Vertx instances continue to attempt to send the request to the old IP address that went away. This is permanent, there is no recovery unless I restart the sender.

melaraj2 avatar Jun 21 '20 03:06 melaraj2

can you provide a reproducer ?

On Sun, Jun 21, 2020 at 5:17 AM Manuel Elaraj [email protected] wrote:

The problem is not just the exception, in my case I have a JVM instance that calls eventBus.request to an address being consumed by a set of JVMs, when I restart the consumers, the requests to that address fail with timeout exceptions. It appears that the only way to resolve it is by also restarting the requester JVMs. In my case, it is a normal course of business to update the consumers in the cluster. I imagine this is a common thing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vert-x3/vertx-zookeeper/issues/63#issuecomment-647073092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCSZSZJHP356KB3YSC3RXV3WHANCNFSM4ECXDK7A .

vietj avatar Jun 21 '20 10:06 vietj