kafka-connect-hdfs

kafka-connect-hdfs copied to clipboard

When i was going through **documentation** of **5.5.0** version it is mentioned that **format.class** property of connector can have **io.confluent.connect.hdfs.orc.OrcFormat** as value. Link : https://docs.confluent.io/5.5.0/connect/kafka-connect-hdfs/configuration_options.html#connector But when i tried using...

vivek43nit

Feature request: Rotation based on maximum file size on hdfs.

8

I'd like the option to specify the maximum file size for the hdfs connector to write before rotating. I understand the only way to do this is to approximate it...

TomLous

HDFS writer can't be restarted correctly after the stopping of a datanode

2

sometimes the HDFS writer can't be restarted correctly when a trouble occurs into the datalake (after the stopping of a datanode) .HDFS logs may be corrupted, already openforwrite and the...

swathimocharla

Kafka Connect HDFS and Azure HDInsights 4

2

We are trying to use Kafka Connect HDFS for Azure HDInsights 4 and Data Lake 2. However, the data lake .jar file requires hadoop-common-3.1.1, where as the downloadable connector only...

richardjg74

Kafka connect task failed with NullPointer Exception

8

java.lang.NullPointerException at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:160) at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:109) at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:302) at io.confluent.connect.hdfs.parquet.ParquetRecordWriterProvider$1.close(ParquetRecordWriterProvider.java:112) at io.confluent.connect.hdfs.TopicPartitionWriter.closeTempFile(TopicPartitionWriter.java:689) at io.confluent.connect.hdfs.TopicPartitionWriter.close(TopicPartitionWriter.java:447) at io.confluent.connect.hdfs.DataWriter.close(DataWriter.java:459) at io.confluent.connect.hdfs.HdfsSinkTask.close(HdfsSinkTask.java:148) at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:396) at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:590) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at...

abhisheksahani

RAM_DISK / LAZY_PERSIST used in kafka connect for commiting files to hdfs is not officially supported by hdfs

5

Hi team, we raised an [issue](https://issues.apache.org/jira/browse/HDFS-14947) related to RAM_DISK / LAZY_PERSIST in which after successfull commit of files also , file was not present in hdfs cluster after rename. we...

abhisheksahani

low rate of consumption in some of partitions in topic

2

Hello team, we are running the hdfs connector with following config: { "name": "hdfs-connector", "config": { "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", "tasks.max": "30", "topics": "kulla_hdfs_test", "hdfs.url": URL, "flush.size": "30000", "rotate.interval.ms" : 10000, "name":...

kullatomer

kafka-connect.service fails with new java version

2

We faced the below issue appears to be due to the ciphers deprecated in the new upgraded java version from 1.8.0.181 to 1.8.0_242 and it looks like a problem in...

amonsef82

Set GZIP compression for Parquet FIle

2

Hello, I have the following configuration for sink connector. Is there any possibility to set a custom compression for Parquet files? By default is Snappy, i would like to change...

zizake

java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy

8

Facing issues when format.class used with parquet data format. Stack trace [2019-03-22 07:45:51,991] INFO Flushing mem columnStore to file. allocated memory: 64 (org.apache.parquet.hadoop.InternalParquetRecordWriter:160) [2019-03-22 07:45:51,991] ERROR Error closing writer for...

kaushiksrinivas