dubbo icon indicating copy to clipboard operation
dubbo copied to clipboard

k8s环境下,provider重新部署后,consumer一直重连老的provider ip

Open boog14 opened this issue 3 years ago • 9 comments

  • [ ] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

  • spring boot: 2.4.10
  • zookeeper: 3.6.4
  • Dubbo version: 3.0.3
  • Java version: 1.8

Steps to reproduce this issue

1、腾讯云k8s上重新部署provider,此时provider pod重建后,ip由192.168.1.207 变成 192.168.1.208 2、consumer的RecollectTimerTask一直尝试连接provider的192.168.1.207

错误信息如下 2021-12-11 22:17:46.137|ERROR|ubbo-client-idleCheck-thread-1|o.a.d.r.e.s.header.ReconnectTimerTask | [DUBBO] Fail to connect to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/192.168.1.77:39932 -> /192.168.1.207:20880]], dubbo version: 3.0.3, current host: 192.168.1.77 org.apache.dubbo.remoting.RemotingException: client(url: dubbo://192.168.1.207:20880/com.fs.service.product.TradeProductService?anyhost=true&application=product-service&check=false&codec=dubbo&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&heartbeat=60000&interface=com.fs.service.product.TradeProductService&metadata-type=remote&methods=approveById,queryByPage,removeById,create,updateById&release=3.0.3&revision=1.0-SNAPSHOT&service-name-mapping=true&service.filter=threadExecutorFilter&side=provider&timeout=6000&token=92f77e4c-37fe-450e-be3f-96e66eb9e71b&version=1.0.0.test) failed to connect to server /192.168.1.207:20880 client-side timeout 3000ms (elapsed: 3001ms) from netty client 192.168.1.77 using dubbo version 3.0.3 at org.apache.dubbo.remoting.transport.netty4.NettyClient.doConnect(NettyClient.java:175) at org.apache.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:202) at org.apache.dubbo.remoting.transport.AbstractClient.reconnect(AbstractClient.java:258) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.reconnect(HeaderExchangeClient.java:166) at org.apache.dubbo.remoting.exchange.support.header.ReconnectTimerTask.doTask(ReconnectTimerTask.java:49) at org.apache.dubbo.remoting.exchange.support.header.AbstractTimerTask.run(AbstractTimerTask.java:87) at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:648) at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:727) at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:449) at java.lang.Thread.run(Thread.java:748)

期望

请问如何解决consumer一直刷Fail to connect to HeaderExchangeClient的问题,在其他issue中也看到很多类似这样的bug但是都没有解决方案,有说3.0.3好使的,但是从3.0.2.1升级到3.0.3还是不好使。

boog14 avatar Dec 11 '21 16:12 boog14

k8s里面不应该连接svc的IP吗,为什么要连接pod的ip?

baymax55 avatar Dec 13 '21 06:12 baymax55

用的什么注册中心配置?需要确认下有没有收到最新的地址通知

chickenlj avatar Dec 16 '21 05:12 chickenlj

我也遇到这个问题,该怎么解决

huanyang001 avatar Dec 27 '21 05:12 huanyang001

用的什么注册中心配置?需要确认下有没有收到最新的地址通知

我也遇到这个问题,用的nacos注册中心配置 服务提供方重新部署后会有短暂的连接之前的ip报错,然后才会切换过来 感觉是不是nacos1.x 不支持长链接 推送失败导致的

yisirman avatar Dec 29 '21 09:12 yisirman

用的什么注册中心配置?需要确认下有没有收到最新的地址通知

我也遇到这个问题,用的nacos注册中心配置 服务提供方重新部署后会有短暂的连接之前的ip报错,然后才会切换过来 感觉是不是nacos1.x 不支持长链接 推送失败导致的 用的nacos 2.0.3,现象就是旧的pod会推送消息给consumer,但是consumer还是连就是pod而不会去连新的pod

huanyang001 avatar Jan 06 '22 07:01 huanyang001

dubbo 2.7.12 nacos:2.x 也会遇到这个问题,当provider部署后,consumer就报错:Host is unreachable,然后重启consumer就好了

rory-zhang avatar Jan 13 '22 05:01 rory-zhang

  • [ ] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

  • spring boot: 2.4.10
  • zookeeper: 3.6.4
  • Dubbo version: 3.0.3
  • Java version: 1.8

Steps to reproduce this issue

1、腾讯云k8s上重新部署provider,此时provider pod重建后,ip由192.168.1.207 变成 192.168.1.208 2、consumer的RecollectTimerTask一直尝试连接provider的192.168.1.207

错误信息如下 2021-12-11 22:17:46.137|ERROR|ubbo-client-idleCheck-thread-1|o.a.d.r.e.s.header.ReconnectTimerTask | [DUBBO] Fail to connect to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/192.168.1.77:39932 -> /192.168.1.207:20880]], dubbo version: 3.0.3, current host: 192.168.1.77 org.apache.dubbo.remoting.RemotingException: client(url: dubbo://192.168.1.207:20880/com.fs.service.product.TradeProductService?anyhost=true&application=product-service&check=false&codec=dubbo&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&heartbeat=60000&interface=com.fs.service.product.TradeProductService&metadata-type=remote&methods=approveById,queryByPage,removeById,create,updateById&release=3.0.3&revision=1.0-SNAPSHOT&service-name-mapping=true&service.filter=threadExecutorFilter&side=provider&timeout=6000&token=92f77e4c-37fe-450e-be3f-96e66eb9e71b&version=1.0.0.test) failed to connect to server /192.168.1.207:20880 client-side timeout 3000ms (elapsed: 3001ms) from netty client 192.168.1.77 using dubbo version 3.0.3 at org.apache.dubbo.remoting.transport.netty4.NettyClient.doConnect(NettyClient.java:175) at org.apache.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:202) at org.apache.dubbo.remoting.transport.AbstractClient.reconnect(AbstractClient.java:258) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.reconnect(HeaderExchangeClient.java:166) at org.apache.dubbo.remoting.exchange.support.header.ReconnectTimerTask.doTask(ReconnectTimerTask.java:49) at org.apache.dubbo.remoting.exchange.support.header.AbstractTimerTask.run(AbstractTimerTask.java:87) at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:648) at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:727) at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:449) at java.lang.Thread.run(Thread.java:748)

期望

请问如何解决consumer一直刷Fail to connect to HeaderExchangeClient的问题,在其他issue中也看到很多类似这样的bug但是都没有解决方案,有说3.0.3好使的,但是从3.0.2.1升级到3.0.3还是不好使。

请问这个问题解决了吗,我也遇到同样的问题

wenlincheng avatar Aug 19 '22 06:08 wenlincheng

3.1.0 仍有同样的问题

liuzhiguo630 avatar Oct 12 '22 03:10 liuzhiguo630

May related with #10570

AlbumenJ avatar Oct 12 '22 03:10 AlbumenJ

dubbo 2.7.12 nacos:2.x 也会遇到这个问题,当provider部署后,consumer就报错:Host is unreachable,然后重启consumer就好了

请问,大家之前的这个问题定位到原因、解决了吗? 感谢

YuWan1117 avatar Oct 25 '22 02:10 YuWan1117

May be fixed in https://github.com/apache/dubbo/pull/10938. Please feel free to reopen or create a new PR if you have any question.

AlbumenJ avatar Nov 24 '22 04:11 AlbumenJ