seatunnel
seatunnel copied to clipboard
apache
[Bug] [Connector-hive-v2] can't get table from hive3.x by the seatunnel 2.3.1-release
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
there are a problem when i start with hive 3.x . i can't get the table from hive although Kerberos authentication successful and connection is opened.
SeaTunnel Version
seatunnel2.3.1 zeta
SeaTunnel Config
standalone
env {
execution.parallelism = 1
job.mode = "BATCH"
job.name = "mysql_hive_test"
}
source {
Hive {
table_name = "seatunnel_test.test_person_yxf_part2"
metastore_uri = "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083"
kerberos_principal = "hive/[email protected]"
kerberos_keytab_path = "/etc/security/keytabs/hive.service.keytab"
hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
}
}
sink {
Console {}
}
Running Command
./bin/seatunnel.sh --config ./config/hive-mysql-tbds.conf -e local
Error Exception
(1):
WARN hive.metastore - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4247) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4233) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:128) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.1]
ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
(2)：
Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60)
at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123)
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
... 2 more
Flink or Spark Version
No response
Java or Scala Version
java8
Screenshots
No response
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
all logs:
JAVA_OPTS: -Dhazelcast.client.config=/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml -Dseatunnel.config=/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml -Dhazelcast.config=/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml -Dlog4j2.configurationFile=/opt/apache-seatunnel-incubating-2.3.1/config/log4j2_client.properties -Dseatunnel.logs.path=/opt/apache-seatunnel-incubating-2.3.1/logs -Dseatunnel.logs.file_name=seatunnel-starter-client-1680141163115 -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-4431960592068079776 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/ 2023-03-30 09:52:43,922 WARN org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator -
-e and --deploy-mode will be deprecated in 2.3.1, please use -m and --master instead of it
三月 30, 2023 9:52:43 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config' 三月 30, 2023 9:52:43 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml 三月 30, 2023 9:52:43 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig 信息: seatunnel.home is /opt/apache-seatunnel-incubating-2.3.1 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config' 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config' 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config' 三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml 2023-03-30 09:52:44,417 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct. 2023-03-30 09:52:44,417 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1] 2023-03-30 09:52:44,418 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] 2023-03-30 09:52:44,441 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start... 2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7) 2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Cluster name: seatunnel-823219 2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1]
/ | | | | |
\ --. ___ __ _ | | _ _ _ __ _ __ ___ | | --. \ / _ \ / ` | | | | | | || ' \ | ' \ / _ | |
/_/ /| __/| (| | | | | || || | | || | | || /| |
_/ _| _,| _/ _,||| |||| || _||_|
2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation. 2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed. To enable integrity checker do one of the following:
- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) 2023-03-30 09:52:44,446 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] The Jet engine is disabled. To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) 2023-03-30 09:52:44,877 INFO com.hazelcast.system.security - [localhost]:5801 [seatunnel-823219] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config. 2023-03-30 09:52:44,941 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-823219] [5.1] Using TCP/IP discovery 2023-03-30 09:52:44,942 WARN com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel-823219] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees. 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config' 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config' 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml 2023-03-30 09:52:45,128 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-823219] [5.1] Created new BusWork : 1730129134 2023-03-30 09:52:45,139 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801 2023-03-30 09:52:45,142 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-823219] [5.1]
CoordinatorService Thread Pool Status
activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0
2023-03-30 09:52:45,203 INFO com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel-823219] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. 2023-03-30 09:52:45,210 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is STARTING 2023-03-30 09:52:45,232 INFO com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel-823219] [5.1]
Members {size:1, ver:1} [ Member [localhost]:5801 - a5d08ac1-a938-4c8a-be7f-4fa206977ab8 this ]
2023-03-30 09:52:45,241 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is STARTED 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml' from System property 'hazelcast.client.config' 三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator 信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml 2023-03-30 09:52:45,270 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel-823219] [5.1] Initializing cluster partition table arrangement... 2023-03-30 09:52:45,474 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-823219] [5.1] Running with 2 response threads, dynamic=true 2023-03-30 09:52:45,485 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING 2023-03-30 09:52:45,486 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED 2023-03-30 09:52:45,492 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Trying to connect to cluster: seatunnel-823219 2023-03-30 09:52:45,493 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Trying to connect to [localhost]:5801 2023-03-30 09:52:45,512 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel-823219] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:47926, qualifier=null, endpoint=[127.0.0.1]:47926, remoteUuid=bea0232d-1875-4bd5-8055-cc54d4d61773, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: bea0232d-1875-4bd5-8055-cc54d4d61773, client name: hz.client_1, client version: 5.1 2023-03-30 09:52:45,514 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED 2023-03-30 09:52:45,514 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Authenticated with server [localhost]:5801:a5d08ac1-a938-4c8a-be7f-4fa206977ab8, server version: 5.1, local address: /127.0.0.1:47926 2023-03-30 09:52:45,515 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-823219] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. 2023-03-30 09:52:45,521 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-823219] [5.1]
Members [1] { Member [localhost]:5801 - a5d08ac1-a938-4c8a-be7f-4fa206977ab8 }
2023-03-30 09:52:45,540 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds. 2023-03-30 09:52:45,576 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[file:/opt/apache-seatunnel-incubating-2.3.1/plugins/jdbc/lib/mysql-connector-j-8.0.31.jar] 2023-03-30 09:52:45,581 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/hive-mysql-tbds.conf 2023-03-30 09:52:45,627 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64] 2023-03-30 09:52:45,669 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: { "env" : { "execution.parallelism" : 1, "job.mode" : "BATCH", "job.name" : "mysql_hive_test" }, "source" : [ { "hdfs_site_path" : "/etc/hadoop/conf/hdfs-site.xml", "kerberos_principal" : "hive/[email protected]", "metastore_uri" : "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083", "plugin_name" : "Hive", "table_name" : "seatunnel_test.test_person_yxf_part2", "kerberos_keytab_path" : "/etc/security/keytabs/hive.service.keytab" } ], "sink" : [ { "plugin_name" : "Console" } ] }
2023-03-30 09:52:45,686 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/hive-mysql-tbds.conf 2023-03-30 09:52:45,689 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: { "env" : { "execution.parallelism" : 1, "job.mode" : "BATCH", "job.name" : "mysql_hive_test" }, "source" : [ { "hdfs_site_path" : "/etc/hadoop/conf/hdfs-site.xml", "kerberos_principal" : "hive/[email protected]", "metastore_uri" : "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083", "plugin_name" : "Hive", "table_name" : "seatunnel_test.test_person_yxf_part2", "kerberos_keytab_path" : "/etc/security/keytabs/hive.service.keytab" } ], "sink" : [ { "plugin_name" : "Console" } ] }
2023-03-30 09:52:45,697 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
2023-03-30 09:52:45,707 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Hive at: file:/opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar
2023-03-30 09:52:45,715 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Hive'} from path: file:/opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
2023-03-30 09:52:45,745 INFO org.apache.hadoop.hive.conf.HiveConf - Found configuration file null
2023-03-30 09:52:46,100 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-03-30 09:52:46,116 INFO org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils - Start Kerberos authentication using principal hive/[email protected] and keytab /etc/security/keytabs/hive.service.keytab
2023-03-30 09:52:46,319 INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user hive/[email protected] using keytab file /etc/security/keytabs/hive.service.keytab
2023-03-30 09:52:46,319 INFO org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils - Kerberos authentication successful
2023-03-30 09:52:46,335 INFO hive.metastore - Trying to connect to metastore with URI thrift://ambari-31.snowleopard.cn:9083
2023-03-30 09:52:46,353 INFO hive.metastore - Opened a connection to metastore, current connections: 1
2023-03-30 09:52:46,431 WARN hive.metastore - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4247) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4233) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
===============================================================================
2023-03-30 09:52:49,600 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,
2023-03-30 09:52:49,600 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2023-03-30 09:52:49,601 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed
2023-03-30 09:52:49,602 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87) at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60) at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) ... 2 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1346) at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:83) ... 11 more
2023-03-30 09:52:49,602 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87) at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60) at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) ... 2 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1346) at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:83) ... 11 more
check u hadoop-client & hive-client version is same with u server version
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
same problem; my versions: hive version is 3.1.0; hadoop version is 3.1.1;
same problem; my versions: hive version is 1.1.0; hadoop version is 2.6.0; seatunnel used hive-exec-2.3.9.jar
same problem; my versions: Hadoop 3.1.2 hive-3.1.3-hadoop3.1-1.0.0 spark-2.4.8-hadoop3.2-1.0.5
seatunnel use hive-exec-3.1.3.jar
Could you please offer more information of your hive metastore server authentication?
I will aim to this problem submit a patch, hey guys please wait my news
same problem; my versions: hive version is 2.1.1; hadoop version is 3.0.0;
seatunnel used hive-exec-2.3.9.jar
