kafka-connect-hdfs

kafka-connect-hdfs copied to clipboard

Recent releases not in confluent repo

8

I can not find recent releases in confluent package repo: http://packages.confluent.io/maven/io/confluent/kafka-connect-hdfs/ Last available is 5.2.4 but latest release is 10.0.0. Is this expected? Is there a way how to consume...

JozoVilcek

HDFS3 sink and Hive Integration Hive 2.1.1 (get_table_req)

1

Using hdfs3 sink with Hadoop 3.0.0 and Hive 2.1.1 Files are written to HDFS, Hive table is created, but adding partitions results in below error It seems method 'get_table_req' does...

ghost

HDFS Sink Connector 5.4.3 NullPointerException when tasks are started

1

I'm using HDFS Sink Connector version 5.4.3 and sometimes (not always) when I start tasks, I get the below exception: ``` org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig logAll - EnrichedConnectorConfig values: config.action.reload = restart connector.class...

smj19

Facing issue while trying to get data from postgres sql into kafka topics using debezium plugin

1

Here is the issue that I am facing: ERROR WorkerSourceTask{id=Trail-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:187) org.apache.kafka.connect.errors.ConnectException:...

Snehil-upes

I have table like this ``` id name year month day emp01 karl 2021 05 01 ``` and use FieldPartitioner, partition.filed.name is "year". data commit succeeded, but when I use...

raybin7

Question about temp files

1

When HDFS Sink connector start buffering records, it writes a temp file at +/tmp//_tmp.json hdfs path. Is HDFS Sink connector writing this temp file every time it recevies a message...

dinegri

Custom output directory naming (without topic in it)

5

Hello, is it possible / safe to provide a custom directory naming convention where one do not want to use actual topic name? I see it possible via custom partitioner...

JozoVilcek

Confluent 5.0.0 : Kafka hdfs sink connector failures when the connector log file being corrupt/another task cannot acquire lease.

5

We upgraded from confluent 5.0.0 from 4.0.0 in production to run kafka connect hdfs connectors? I see connector task failures with the below error. I think this issue is related...

rupeshmore85

java.lang.RuntimeException: org.apache.kafka.connect.errors.SchemaProjectorException: Schema parameters not equal

4

We are using HDFS connector v4.0.1 and Kafka Connect 1.0.0 Some data is already written into HDFS in parquet format. The associated schema contains "redacted comment v1" in doc field....

rohitkug

it will create many small files?

1

Hi, everyone , recently i used hdfs sink connector, and i am worried about that if the data is consistenctly input to kafka, will it generate some small files, which...

katty0924