kafka-connect-hdfs

kafka-connect-hdfs copied to clipboard

Issue when kafka connect worker pod restart, during creation of nested partition directories in hdfs file system.

3

In our production labs, an issue is observed. Below is the sequence of the same. 1. hdfs connector is added to the connect worker. 2. hdfs connector is creating folders...

kaushiksrinivas

Rotate Interval doesn't work for low volume or irregular traffic

5

I was looking at the logs on our QA environment and got very concerned about using it in production, because the rotate interval configurations (both `rotate.interval.ms` and `rotate.schedule.interval.ms`) do not...

dcsobral

Data is Hive table shows new columns as NULL after schema evolution with a default value specified when the format is Parquet

12

hi there, I have avro data that is being written to HDFS with hive integration turned on with `"format.class":"io.confluent.connect.hdfs.parquet.ParquetFormat"` and `"schema.compatibility": "BACKWARD"`. Add a new column to the schema with...

swathimocharla

Hi, was having issue running the connector with the following error: ``` org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at...

ErwinLee10

Kafka connector Hive integration issue

2

Hi, I have integrated Kafka and Hadoop. I am successfully writing data to Hadoop. However, when I open the hive integration, I get the following error. I couldn't find the...

gokhandroid

Exactly-once should also work without topic name included in the path

2

This change is necessary to retain EOS semantics to support flexible storage partitioning scheme as proposed here: https://github.com/confluentinc/kafka-connect-storage-common/issues/126

hariprasad-k

Support hadoop.proxyusr in connector configuration

1

blcksrx

I was trying to convert avro files to parquet and sink them by using kafka-connect-hdfs connector. According to this [confluent blog](https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/), It is possible to go with multiple schema for...

gokhansari

Fix the data loss problem

6

## Problem When HDFS fails, the data being written will be lost. ## Solution it is necessary to record the offset. When HDFS is recovered, the offset will be reset...

lightzhao

HCD-481 : Adding reconnection logic added for HDFS2 sink connector

2

## Problem The connector was failing for `Avro` and `ORC` format when the connection to the cluster was interrupted. ## Solution **Fix for Avro** - The schema was re-initialized to...

SanchayGupta1197