logging-flume icon indicating copy to clipboard operation
logging-flume copied to clipboard

FLUME-3342:Fix the taildir source read the data duplication.

Open zhulh200868 opened this issue 5 years ago • 2 comments

If the file a.log reaches a certain size, it will be renamed a.log.1 and a new file a.log, then a.log and a.log.1 inode are the same, but the file name is different, the default flume taildir source will think that this is a new file, it will be re-read, that causing data duplication.

zhulh200868 avatar Jun 29 '19 13:06 zhulh200868

This is not a taildirsource rule.

xiaoqingwuku avatar Jul 24 '19 10:07 xiaoqingwuku

This can be handled by filename regex in flume.conf. If the filename regex is modified not to include a renamed file, this problem can be solved.

iijima-satoshi avatar May 11 '22 12:05 iijima-satoshi