kafka-connect-hdfs

kafka-connect-hdfs copied to clipboard

Task failed with NullPointerException

7

I got the following issue:  "trace": "java.lang.NullPointerException\n\tat io.confluent.connect.hdfs.HdfsSinkTask.open(HdfsSinkTask.java:143)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.openPartitions(WorkerSinkTask.java:586)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.access$1100(WorkerSinkTask.java:67)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask$HandleRebalance.onPartitionsAssigned(WorkerSinkTask.java:651)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:285)\n\tat org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:424)\n\tat org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:443)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:316)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"

huynhphuong10284

Connecting to Kerberos HA cluster

3

While connecting to kerberos HA enabled cluster getting the error " Server has invalid Kerberos principal: hdfs/namenode_hostname@CORP" I have configured below kerberos config for hdfs connector hdfs.authentication.kerberos=true connect.hdfs.principal=abc connect.hdfs.keytab=/etc/security/keytabs/abc.keytab hdfs.namenode.principal=hdfs/hdfs_namespace@CORP...

lijubjohn

Kafka hdfs connect exception when datanode is restarted

2

Hi, I have a single HDFS writer that has 1 task. This is writing to HDFS with a single datanode. When the datanode goes for a restart, we see the...

swathimocharla

org.apache.kafka.connect.errors.ConnectException: java.io.IOException: The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: COLUMN

6

org.apache.kafka.connect.errors.ConnectException: java.io.IOException: The file being written is in an invalid state. Probably caused by an error thrown previously. Current state: COLUMN at io.confluent.connect.hdfs.parquet.ParquetRecordWriterProvider$1.close(ParquetRecordWriterProvider.java:97) at io.confluent.connect.hdfs.TopicPartitionWriter.closeTempFile(TopicPartitionWriter.java:647) at io.confluent.connect.hdfs.TopicPartitionWriter.closeTempFile(TopicPartitionWriter.java:654) at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:386) at...

wanygan83

Cannot get out of Exception when closing the temporary file has failed

4

Hello, For some reason, when writing the parquet temporary file into HDFS filesystem,we got this exception: 2019/01/18 14:40:05 ERROR 2525665 [pool-3-thread-28] connect.hdfs.TopicPartitionWriter.write(403): Exception on topic partition SYS_4G_PCMD_IMSI_PROC-7: org.apache.kafka.connect.errors.ConnectException: java.io.IOException: can...

User1773

Hdfs sink was working fine for sometime but now my sink is not working

4

I am trying to park data in HAdoop in real time from Mysql using confluent-5.0.1. My hdfs-sing is throwing error with mentioned below. Mysql jdbc connection: { "name": "jdbc_source_mysql_registration_query", "config":...

bkotesh

in both AvroUtils.java and ParquetUtils.java. the `alterTable` are called with only "table" parameter. we need to support "Cascade" due to table schema evolution. ``` @Override public void alterSchema( String database,...

ericjin527

modify COMMMITTED_FILENAME_SEPARATOR to under line

8

If filename in HDFS with plus(+) , it can not be viewed at HUE. For example, if filename is **000010+000020.avro** , then you can not get file from brower caused...

yudong2015

We, as developers, are free to define` io.confluent.connect.storage.partitioner.Partitioner ` interface. There is a problem though if `generatePartitionedPath` generates path that starts with directory named after the topic. https://github.com/confluentinc/kafka-connect-hdfs/blob/62fd4a5bd0c5e00f9e68a34851ab18cc0f4b9eda/src/main/java/io/confluent/connect/hdfs/TopicPartitionWriter.java#L563 This particular...

bottaio

Exceptions when network is broken

9

I have a hdfs connector running on my machine, which reads from a Kafka Cluster and then writes into a HDFS cluster. When I disable my machine's network connection, and...

skyahead