How to use Kerberos in Hudi integrated Flink?data cannot be written to hdfs.

[FWLamb]

I have configured the Kerberos-related parameters in the Flink configuration file and executed the kinit command before entering SQL client, but it still failed.

Environment Description

• Hudi version : 0.14.0

• Flink version : 1.16.2

• Hadoop version :3.1.0

• Storage (HDFS/S3/GCS..) : hdfs

[danny0405]

Did you config the kerb certificate in you flink configurations?

[FWLamb]

Did you config the kerb certificate in you flink configurations?

Yes, please take a look at the fourth picture above.

[danny0405]

Did you configure it on all the cluster nodes?

[FWLamb]

Did you configure it on all the cluster nodes?

I only deployed a single node for testing

[FWLamb]

When I submit a job by writing code, it can be successful, but this problem arises when using SQL client.

[danny0405]

SQL client has it's own config yamls, not sure whether we can configure the kerb there

[FWLamb]

Flink1.14 and later versions have removed the sql-client-defaults. yaml file

[jiaojietao]

Can you write the group:user to which the path "tmp/hudi_flink/t1" belongs? Can you take a screenshot and see if your ticket should be configured as' hive:hive '

[Toroidals]

Translation: I also encountered the same problem. When integrating Flink with Hudi, I created a table in Flink-SQL and synchronized it to Hive. However, I cannot perform Hive JDBC Kerberos authentication during the synchronization to Hive. My Hive cluster does not have user passwords, only Kerberos authentication.

[danny0405]

Did you try the HMS sync mode then instead of the JDBC.

[wangzhenwen681]

I have also encountered this problem, and I would like to ask you how you can solve it

[danny0405]

How did you configure the kerb certificate in your hive conf?

[wangzhenwen681]

jobManager log 2023-12-22 16:35:07,730 WARN org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it. org.apache.thrift.transport.TTransportException: null hivemetastore log ERROR org.apache.thrift.server.TThreadPoolServer - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status -128 Caused by: org.apache.thrift.transport.TTransportException: Invalid status -128 environment hudi-0.14.0 flink-1.16.2 flinksql is CREATE TABLE test_hudi(

id int, num int, ts int, primary key (id) not enforced ) PARTITIONED BY (num) with( 'connector'='hudi', 'path' = 'hdfs:///data/test/test_hudi', 'table.type'='COPY_ON_WRITE',
'hive_sync.enable'='true', 'hive_sync.db'='test', 'hive_sync.table'='test_hudi', 'hive_sync.mode'='hms' ,
'hive_sync.metastore.uris' = 'thrift://datasophon01:9083', 'hive_sync.use_jdbc'='true', 'hive_sync.use_kerberos' = 'true', 'hive_sync.kerberos.krb5.conf' = '/etc/krb5.conf', 'hive_sync.kerberos.principal' = 'hive/[email protected]', 'hive_sync.kerberos.keytab.file' = '/etc/security/keytab/hive.service.keytab', 'hive_sync.kerberos.keytab.name' = 'hive/[email protected]'
);

[ad1happy2go]

There was an another issue which a user fixed like this - https://github.com/apache/hudi/issues/9269#issuecomment-1659637537

If it is related and works, we can implement it as a general solution. cc @danny0405