kafka-connect-hdfs icon indicating copy to clipboard operation
kafka-connect-hdfs copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

Issue in a Kerberized environment a day after renew ticket

Open dinegri opened this issue 4 years ago • 2 comments

Hi,

I have a problem in a Kerberized environment: when I start the connector everything is working fine, I obtain my Kerberos credentials and the connector start writing without issues. The problem begin when a day after the ticket from Kerberos is renewed and the connector crash immediatly with this error:

ERROR Recovery failed at state RECOVERY_PARTITION_PAUSED (io.confluent.connect.hdfs.TopicPartitionWriter:221) org.apache.kafka.connect.errors.ConnectException: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "abc/10.72.176.21"; destination host is: "abc":8020; at io.confluent.connect.hdfs.wal.FSWAL.apply(FSWAL.java:131) at io.confluent.connect.hdfs.TopicPartitionWriter.applyWAL(TopicPartitionWriter.java:519) at io.confluent.connect.hdfs.TopicPartitionWriter.recover(TopicPartitionWriter.java:204) at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:234) at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:234) at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:91) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:287) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:176) at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90) at org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58) at org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)

I am using Java 11 e kafka-connect-hdfs version 10.0.0

dinegri avatar Sep 22 '21 19:09 dinegri

After digging a little bit more I have found this here in the issues: https://issues.apache.org/jira/browse/HDFS-16165

I will migrate for HDFS Sink 3 once we have a annual contract with Confluent

After migration I will update here if this solution worked

dinegri avatar Sep 22 '21 21:09 dinegri

HDFS Sink 3 does not fix this issue. Is is available only using a commercial license

Following instruction here https://github.com/confluentinc/kafka-connect-hdfs/issues/225 fix the issue

dinegri avatar Dec 01 '21 15:12 dinegri