kafka-connect-spooldir

kafka-connect-spooldir copied to clipboard

Library version: 2.0.64 Connector configuration: ```json { "connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector", "topic": "transactions_spooldir_15", "input.path": "/data/unprocessed", "finished.path": "/data/processed", "error.path": "/data/error", "input.file.pattern": ".*\\.csv", "schema.generation.enabled":"true", "csv.first.row.as.header":"true", "transforms":"castTypes,castDate", "transforms.castTypes.type":"org.apache.kafka.connect.transforms.Cast$Value", "transforms.castTypes.spec":"outflow:float32,inflow:float32,cleared:boolean", "transforms.castDate.type":"org.apache.kafka.connect.transforms.TimestampConverter$Value", "transforms.castDate.target.type": "Timestamp", "transforms.castDate.field": "date", "transforms.castDate.format":...

pcasaretto

[spool-schema-value.txt](https://github.com/jcustenborder/kafka-connect-spooldir/files/9049765/spool-schema-value.txt) I have given the value for this property "schema.generation.value.name": "nz.co.3es.example.customer.Value", and we have given the schema registry info in the worker level. However when I look at the schema,...

esquare666

Incorrect skipping of records when reprocessing file

2

The CSV connector is incorrectly under-skipping records when reprocessing a file. Assume there is a CSV with 1 header record and 10 data records named EMPS.csv When the connector processes...

chipps

Continious logs coming in kafka connect logs

2

No files matching input.file.pattern were found in /data/unprocessed/kdg/order_data continious logs are coming when no files exist in the unprocessed folder. Please help

sachin-rajwade

Could not parse <field> to 'Date'

4

Hi, I am using v2.0.62 of the library with Kafka connect 6.1.0. I am trying to parse CSV with values as: `abcd, 2022-01-10T23:09:55.000Z, abcdef, ....` Using this schema ``` {...

acamillo

Hi. I am using the SpoolDirCsvSourceConnector to place records from a file onto a topic. I have a JAVA app that will consume these records. I'm looking for a way...

ronb722

While fetching JSON data , shows integer value to null

2

Hi, i tried to fetch data from json file through "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirJsonSourceConnector". the problem I encountered is that the integer value and float value are displayed null in the topic other...

dahal4

Notifications in a distinct topic when starting/ending each file

1

As it has been asked a couple of times, here is a proposal to implement notifications when files are processed. The basic idea is to add some `SourceRecord` during the...

bleporini

(com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnectorConfig:354) kafka-connect | [2021-12-23 02:18:29,369] ERROR [NCS_FS_PRODUCT_MARGIN_SOURCE|worker] WorkerConnector{id=NCS_FS_PRODUCT_MARGIN_SOURCE} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:193) kafka-connect | java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkState(boolean, java.lang.String, java.lang.Object)' kafka-connect | at com.github.jcustenborder.kafka.connect.utils.config.ConfigUtils.getEnums(ConfigUtils.java:82) kafka-connect | at com.github.jcustenborder.kafka.connect.spooldir.AbstractSourceConnectorConfig.(AbstractSourceConnectorConfig.java:164) kafka-connect |...

sachin-rajwade

CSV connector's task hanged in infinite loop after meeting record it can't process

1

Hello! We're testing SpoolDirCsvSourceConnector and faced following condition: 1. Part of file was processed (2000 rows) 2. Error from log: [2021-08-24 18:55:46,547] ERROR Exception encountered processing line 5692 of /connect_integrations/848_quality/DEV/SWAP/SWAP-2021_06_0001_0.csv....

MaxSamsonov