cachecloud icon indicating copy to clipboard operation
cachecloud copied to clipboard

Published 20 hours ago • verified publisher icon sohutv

部署redis-cluster时：ERR Unknown node ****

Open csspark opened this issue 7 years ago • 4 comments

2017-08-10 17:12:49.467 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - runInstance-fallback : redis-cli -h ... -p 6386 shutdown 2017-08-10 17:12:49.507 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6381 isSingleNode 2017-08-10 17:12:49.580 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6382 isSingleNode 2017-08-10 17:12:49.636 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6383 isSingleNode 2017-08-10 17:12:49.699 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6384 isSingleNode 2017-08-10 17:12:49.764 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6385 isSingleNode 2017-08-10 17:12:49.818 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6386 isSingleNode 2017-08-10 17:12:52.389 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - ERR Unknown node 9168cd159e927816698db9effbe7f04c07fcc073 redis.clients.jedis.exceptions.JedisDataException: ERR Unknown node 9168cd159e927816698db9effbe7f04c07fcc073 at redis.clients.jedis.Protocol.processError(Protocol.java:118) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at redis.clients.jedis.Protocol.process(Protocol.java:152) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at redis.clients.jedis.Protocol.read(Protocol.java:206) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:282) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:208) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at redis.clients.jedis.Jedis.clusterReplicate(Jedis.java:3343) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na] at com.sohu.cache.redis.impl.RedisDeployCenterImpl$4.execute(RedisDeployCenterImpl.java:241) ~[classes/:na] at com.sohu.cache.util.IdempotentConfirmer.run(IdempotentConfirmer.java:27) [classes/:na] at com.sohu.cache.redis.impl.RedisDeployCenterImpl.startCluster(RedisDeployCenterImpl.java:257) [classes/:na] at com.sohu.cache.redis.impl.RedisDeployCenterImpl.deployClusterInstance(RedisDeployCenterImpl.java:104) [classes/:na] at com.sohu.cache.stats.app.impl.AppDeployCenterImpl.deployCluster(AppDeployCenterImpl.java:447) [classes/:na] at com.sohu.cache.stats.app.impl.AppDeployCenterImpl.allocateResourceApp(AppDeployCenterImpl.java:334) [classes/:na] at com.sohu.cache.web.controller.AppManageController.doAddAppDeploy(AppManageController.java:477) [classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_141] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:860) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:243) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-embed-core-8.0.30.jar:8.0.30] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.30.jar:8.0.30] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141] ......

2017-08-10 17:13:02.446 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6382 set replicate:{} 2017-08-10 17:13:02.446 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - startCluster create error!

csspark avatar Aug 10 '17 09:08 csspark

新建集群时出现Unknown node，一般是cluster节点meet通信失败或者异步操作延迟导致，请在节点上执行cluster nodes看下这些Unknown 节点的nodeId是否已经存在。如果不存在请检查redis日志。通常是网络防火墙限制导致。

geekyijun avatar Aug 16 '17 09:08 geekyijun

新建集群时出现Unknown node，一般是cluster节点meet通信失败或者异步操作延迟导致，请在节点上执行cluster nodes看下这些Unknown 节点的nodeId是否已经存在。如果不存在请检查redis日志。通常是网络防火墙限制导致。

cluster nodes查看是有对应的nodeId的，但是还是包该错误

xianwei avatar Mar 07 '19 07:03 xianwei

我也遇到这个问题

  1. 防火墙已关闭
  2. 用cluster nodes也有对应nodeId
  3. 直接在机器上操作redis连接都是通的，请问这个有解决办法了没有？

elemman avatar Mar 26 '20 05:03 elemman

我也是这种情况，在meet时只有在自己的nodes.conf中有记录写进来，其他节点的都没有，请问这是正常的吗？

wuwo1952368901 avatar Nov 09 '20 10:11 wuwo1952368901

如遇到该问题，请确认如下信息，如检查后，仍有问题，可reopen或者重新提issue。

  1. 在redis实例上，执行 cluster nodes 确认是否有该nodeId；
  2. redis 实例间网络是否连通，是否被防火墙拦截；
  3. 如网络确认正常，可重试部署任务，查看是否能够部署成功。

另，如果网络存在延迟或部署的第一次报上述错误，可以通过查看最终是否部署成功。默认上述异常对应的命令存在重试机制，如果因为网络延迟，可能会报异常，但如重试成功，不影响最终部署。

githubname1024 avatar Aug 15 '22 03:08 githubname1024