logging-flume
logging-flume copied to clipboard
FLUME-3342:Fix the taildir source read the data duplication.
If the file a.log reaches a certain size, it will be renamed a.log.1 and a new file a.log, then a.log and a.log.1 inode are the same, but the file name is different, the default flume taildir source will think that this is a new file, it will be re-read, that causing data duplication.
This is not a taildirsource rule.
This can be handled by filename regex in flume.conf. If the filename regex is modified not to include a renamed file, this problem can be solved.