kafka-connect-hdfs
kafka-connect-hdfs copied to clipboard
confluentinc
Connecting to Kerberos HA cluster
While connecting to kerberos HA enabled cluster getting the error " Server has invalid Kerberos principal: hdfs/namenode_hostname@CORP" I have configured below kerberos config for hdfs connector
hdfs.authentication.kerberos=true connect.hdfs.principal=abc connect.hdfs.keytab=/etc/security/keytabs/abc.keytab hdfs.namenode.principal=hdfs/hdfs_namespace@CORP
when hdfs.namenode.principal is configured with the active name node , it works fine , but giving the namespace it errors out with invalid server kerberos principal. How to do we connect to HA enabled cluster without hardcoding the namenode?
How to do we connect to HA enabled cluster without hardcoding the namenode?
https://github.com/confluentinc/kafka-connect-hdfs/issues/288#issuecomment-364617889
You may want to copy your existing HDFS client's HADOOP_CONF_DIR into your connect workers, and set hadoop.conf.dir to be that folder.
For example, some have reported a configuration section such as this to work fine.
"hdfs.authentication.kerberos": "true",
"hdfs.namenode.principal": "...@...",
"connect.hdfs.principal": "...@...",
"connect.hdfs.keytab": "/etc/security/keytabs/hdfs.headless.keytab",
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanism": "GSSAPI",
"sasl.kerberos.service.name": "...",
"hadoop.conf.dir": "/etc/hadoop/conf"
Any other Kerberos errors are potentially network related.
Need similar information. so what should be the way to configure hdfs.namenode.principal and connect.hdfs.principal ? What if name node switch happens ? when HA is enabled.
Hdfs sink connector doc says Currently, the connector requires that the principal and the keytab path to be the same on all the hosts running the connector. The host part of the hdfs.namenode.prinicipal needs to be the actual FQDN of the Namenode host instead of the _HOST placeholder.
Confused on this configuration.
Any update of how to configure the namenode.principal in an HA setup?
