kafka-connect-hdfs

kafka-connect-hdfs copied to clipboard

Connector is losing records

3

We just completed configuration and deployment of two Kafka Connect workers with HDFS connectors and we left them running side by side with our current consumer tool (Gobblin) for 3...

alexnu

Please describe "store.url" parameter.

8

So, in documentation on HDFS sink connector we have "The HDFS connection URL. This configuration has the format of hdfs:://hostname:port and specifies the HDFS to export data to. This property...

glarus089

fix for confluent connect hdfs issue 453

8

fix for #453 :Kafka hdfs Connect unable to write to hdfs when it encounters an empty parquet file

swathimocharla19

New columns added to the avro records does not get written to hdfs with schema compatibility as FORWARD.

1

Below is my test scenario, 1. add the connector with below config, `{ "name": "connector2", "config": { "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", "tasks.max": "1", "topics": "apple2", "hdfs.url": "hdfs://xxxx:8020/user/cloud-user/performance/kconnect", "flush.size":"5", "offset.flush.interval.ms":"1000", "offset.flush.timeout.ms":"30000", "format.class":"io.confluent.connect.hdfs.avro.AvroFormat", "consumer.max.partition.fetch.bytes":"70000000",...

kaushiksrinivas

Unknown CI error in #459

1

Pr #459 tries to fix issue #453 . Travis CI states that the commit can not be built, but the build logs are inaccessible.

CsatariGergely

committed parquet files missing from hdfs cluster(Not considering false response while committing file to hdfs)

5

Hi we have seen an issue, where while committing parquet file to hdfs, when flush size or rotation time is reached,connector commit the files to hdfs but doesn't consider the...

abhisheksahani

Enable hive integration (table creation) when data are already present on HDFS

12

Hi all, I would like to enable the Hive integration when there is already some data that have been dumped to HDFS. When there is no data present, the creation...

RossierFl

Caused by: InvalidObjectException(message:default.test_hdfs table not found)

1

after enable hive intergration hive.integration=true, Caused by: InvalidObjectException(message:default.test_hdfs table not found) version: v5.3.1

mylittlecabin

Connector fails when using nested arrays in Avro schema

5

The HDFS connector (version 4.1) fails after the first batch of events when writing _Avro_ messages to _Parquet_ files in HDFS in case there are nested arrays in the Avro...

jurgispods

Avro data in kafka to String data format in hdfs fails with java.lang.ClassCastException: org.apache.kafka.connect.data.Struct cannot be cast to java.lang.String

2

When data is produced to kafka with avro encoding, avro converter used in kafka connect hdfs sink writer fails to write data to hdfs in string format with ClassCastException. Stack...

kaushiksrinivas