kafka-connect-file-pulse icon indicating copy to clipboard operation
kafka-connect-file-pulse copied to clipboard

feat(plugin): Delete the compressed file after extraction

Open rouellet99 opened this issue 10 months ago • 0 comments

feat: (plugin): Add a configuration to delete the compressed file after extraction

Add the flag fs.delete.compress.files.enabled indicating whether compressed file should be deleted after extraction. This flag is added to avoid an infinit loop when using a compressed file with the following configuration: "offset.attributes.string": "name+lastModified", "fs.cleanup.policy.class": "io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanupPolicy", Im fact, if the compressed file is not removed, once extracted, the extracted file is processed, and move to the preconfigured folder. Therefore, the compressed file will be extrated again with a new lastModified value, and the extracted file would be processed again.

When using zipped file, is there any reason , why the zipped file is not removed from the directoty once unzipped? #603

rouellet99 avatar Mar 27 '24 13:03 rouellet99