kyuubi
kyuubi copied to clipboard
Published
20 hours ago •
apache
apache
[Bug] Kyuubi failed to start Spark onYARN
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Search before asking
- [X] I have searched in the issues and found no similar issues.
Describe the bug
Kyuubi failed to start Spark onYARN ，Sometimes it's there and sometimes it's normal，
Zookeeper connection timed out
Affects Version(s)
1.5.2
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
48066 [Driver] ERROR org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection timed out for connection string (emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181) and timeout (15000) / elapsed (31107)
org.apache.kyuubi.shade.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.kyuubi.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
48251 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
48351 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
48452 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49553 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49653 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49754 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
50854 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
50955 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
51055 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52156 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52256 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52357 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53457 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53558 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53658 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54759 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54859 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54960 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56061 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56161 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56262 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57362 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57463 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57563 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58664 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58764 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58865 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
59965 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
60066 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
60167 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61267 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61368 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61468 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62569 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62669 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62770 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
63870 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
63971 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
64069 [Driver] ERROR org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection timed out for connection string (emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181) and timeout (15000) / elapsed (47110)
org.apache.kyuubi.shade.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.kyuubi.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
64071 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65172 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65272 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65373 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66473 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66574 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66674 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67775 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67875 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67976 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69076 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69176 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69277 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70378 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70478 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70579 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71679 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71780 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71880 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
72981 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
73081 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
73182 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74282 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74383 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74483 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75584 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75685 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75785 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
76886 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
76986 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
77087 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78187 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78288 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78388 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79489 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79590 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79690 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
80072 [Driver] WARN org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection attempt unsuccessful after 63113 (greater than max timeout of 60000). Resetting connection and trying again with a new connection.
80791 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send thread for session 0x0.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
80897 [Driver] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
80897 [Driver-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
80898 [Driver] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@4bba1391
80898 [Driver] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
80898 [Driver] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
80899 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
81000 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
81001 [Driver] ERROR org.apache.kyuubi.engine.spark.SparkTBinaryFrontendService - Error starting service EngineServiceDiscovery
org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:178)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1734)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
... 28 more
81003 [Driver] INFO org.apache.kyuubi.engine.spark.SparkTBinaryFrontendService - SparkTBinaryFrontend has stopped
81003 [Driver] ERROR org.apache.kyuubi.engine.spark.SparkSQLEngine - Error starting service SparkTBinaryFrontend
org.apache.kyuubi.KyuubiException: Failed to Start SparkTBinaryFrontend
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:52)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:178)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
... 25 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1734)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
... 28 more
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Hello @wangfeigithub, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Your error message indicates that the root cause is "Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'", please check the Zookeeper log what happens.
Sometimes there are mistakes, sometimes there are no mistakes. And I changed another Zookeeper also occasionally appeared like this
Container from a bad node Exit code :50 is displayed when the SQL is executed. So what does this look like
Container from a bad node Exit code :50 is displayed when the SQL is executed. So what does this look like
sounds like another issue
Sometimes there are mistakes, sometimes there are no mistakes. And I changed another Zookeeper also occasionally appeared like this
what's the error stacktrace on zk? the client-side error indicates it's a network issue.
Zk has no error and I tried two ZKS. The error on the YARN page above will occasionally occur. Nor should the Internet. Using Ali's EMR
I wonder why it occasionally happens.------------------- java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC ---Could this mistake be the root cause? What causes it?
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
I think it was caused by your Spark application classpath jars conflicts.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
I think it was caused by your Spark application classpath jars conflicts.
Maybe there is a conflict with org.slf4j. I will try to remove it to see the Zookeeper problem