moonbox
moonbox copied to clipboard
running-elephant
报错： ERROR spark.Main: Error in configuring object
查询mount的Hive表时，出现ERROR spark.Main: Error in configuring object 这种错误，请问是什么原因造成的？是spark配置不正确吗？？？
20/07/21 15:36:36 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2f3afa0{/metrics/json,null,AVAILABLE,@Spark} 20/07/21 15:36:41 INFO scheduler.EventLoggingListener: Logging events to hdfs://hadoop105:9000/spark22/application_1595310524469_0021 20/07/21 15:36:41 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms) [WARN] [SECURITY][07/21/2020 15:36:42.865] [Moonbox-akka.remote.default-remote-dispatcher-14] [akka.serialization.Serialization(akka://Moonbox)] Using the default Java serializer for class [moonbox.grid.deploy.DeployMessages$RegisterApplication] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [WARN] [SECURITY][07/21/2020 15:38:23.886] [Moonbox-akka.remote.default-remote-dispatcher-5] [akka.serialization.Serialization(akka://Moonbox)] Using the default Java serializer for class [moonbox.grid.deploy.messages.Message$OpenSessionResponse] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' [WARN] [SECURITY][07/21/2020 15:38:53.279] [Moonbox-akka.remote.default-remote-dispatcher-14] [akka.serialization.Serialization(akka://Moonbox)] Using the default Java serializer for class [moonbox.grid.deploy.messages.Message$JobQueryResponse] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage' 20/07/21 15:39:05 ERROR spark.Main: Error in configuring object
以下是我的配置文件 moonbox { deploy { catalog { implementation = "mysql" url = "jdbc:mysql://hadoop105:3306/moonbox?createDatabaseIfNotExist=true" user = "root" password = "123456" driver = "com.mysql.jdbc.Driver" } rest { enable = true port = 9099 request.timeout = "600s" idle.timeout= "600s" } tcp { enable = true port = 10010 } } mixcal { pushdown.enable = true column.permission.enable = true spark.sql.cbo.enabled = true spark.sql.constraintPropagation.enabled = false
cluster = [{
spark.hadoop.yarn.resourcemanager.hostname = "hadoop106"
spark.hadoop.yarn.resourcemanager.address = "hadoop106:8032"
spark.yarn.stagingDir = "hdfs://hadoop105:9000/user/ayy"
spark.yarn.access.namenodes = "hdfs://hadoop105:9000"
spark.loglevel = "ERROR"
spark.cores.max = 1
spark.yarn.am.memory = "512m"
spark.yarn.am.cores = 1
spark.executor.instances = 1
spark.executor.cores = 1
spark.executor.memory = "512m"
}]
} }
好像一执行spark就出现这个问题 我附上spark的配置文件：
------------------------------------------spark-default---------------------------- spark.eventLog.enabled true spark.eventLog.dir hdfs://hadoop105:9000/spark22
spark.yarn.historyServer.address=hadoop105:18080 spark.history.ui.port=18080
「20/07/21 15:39:05 ERROR spark.Main: Error in configuring object」
下面没有详细信息了吗，你用的moonbox什么版本，spark什么版本？