kafka-connect-fs
kafka-connect-fs copied to clipboard
Question - File is not getting moved /delete after processing
I am trying to move the csv file to another folder after processing is done. I can see it being processed to kafka topic but the file is not moved to the given path. Also delete
is not working.
Please find my config below
{
"name": "csv-source-second-conn8",
"connector.class": "com.github.mmolimar.kafka.connect.fs.FsSourceConnector",
"tasks.max": "1",
"fs.uris": "file:///data/unprocessed",
"topic": "csv4",
"policy.class": "com.github.mmolimar.kafka.connect.fs.policy.SimplePolicy",
"policy.recursive": "true",
"policy.regexp": ".*",
"policy.batch_size": "0",
"policy.cleanup": "delete",
"file_reader.class": "com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader",
"file_reader.batch_size": "0",
"file_reader.delimited.settings.header": "true",
"file_reader.delimited.settings.line_separator_detection": "true",
"file_reader.delimited.settings.delimiter_detection": "true"
}
{
"name": "csv-source-second-conn7",
"connector.class": "com.github.mmolimar.kafka.connect.fs.FsSourceConnector",
"tasks.max": "1",
"fs.uris": "file:///data/unprocessed",
"topic": "csv4",
"policy.class": "com.github.mmolimar.kafka.connect.fs.policy.SimplePolicy",
"policy.recursive": "true",
"policy.regexp": ".*",
"policy.batch_size": "0",
"policy.cleanup": "move",
"policy.cleanup.move": "file:///data/processed",
"file_reader.class": "com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader",
"file_reader.batch_size": "0",
"file_reader.delimited.settings.header": "true",
"file_reader.delimited.settings.line_separator_detection": "true",
"file_reader.delimited.settings.delimiter_detection": "true"
}
Please suggest if anything needs to be modified.
@mmolimar - Could you please suggest ?