in_tail: possibly collects duplicated logs in `rotate_wait`

[daipom]

Describe the bug

I think the major log duplication problem about follow_inodes is fixed in #4237. However, there is still a possibility of duplication (https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632).

in_tail possibly collects duplicated logs in rotate_wait. Mulitple TailWatchers can exist in rotate_wait, so they possibly collect the same target in that interval.

We need the mechanism not to add duplicated TailWathcer while detaching it.

I also wrote the code comment as follows in #4237:

https://github.com/fluent/fluentd/blob/d5685ada81ac89a35a79965f1e94bbe5952a5d3a/lib/fluent/plugin/in_tail.rb#L507-L515

This is not something like that all logs in one file are entirely duplicated. (I believe such problems were resolved in #4237)

---

Detail explanation (from https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632)

The scenario: rotate -> update_watcher -> refresh_watcher -> rotate_wait elapsed

• Initial state

path-1: inode-b, TailWatcher-1 
path-2: inode-a, TailWathcer-2

• Rotate happens

path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new, TailWatcher-1-detaching
path-3: inode-a, TailWatcher-2-detaching

• refresh_watcher

path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new, TailWatcher-1-detaching
path-3: inode-a, TailWatcher-2-detaching, TailWatcher-3

• rotate_wait elapsed

path-1: inode-c, TailWatcher-1-new
path-2: inode-b, TailWathcer-2-new
path-3: inode-a, TailWatcher-3

As above, rotate_wait causes the situation that multiple TailWatchers exist for the same target temporarily. This causes log duplication. I have confirmed this on v1.16.1.

To Reproduce

From https://github.com/fluent/fluentd/pull/4237#issuecomment-1633358632.

Use the config below.

• Start to tail the following file

test.log

• Rotate

test.log
test.log.1

• Append a log to test.log.1.
  • When refresh_interval NOT elapsed and rotate_wait NOT elapsed
    • No duplication
  • When refresh_interval elapsed and rotate_wait NOT elapsed
    • DUPLICATION
  • When refresh_interval elapsed and rotate_wait elapsed
    • No duplication

Expected behavior

Log duplication does not occur.

Your Environment

- Fluentd version: 1.16.1, 1.16.2
- Operating system: Ubuntu 20.04.6 LTS
- Kernel version: 5.15.0-71-generic

Your Configuration

<source>
  @type tail
  tag test
  path /path/to/test.log*
  pos_file /test/fluentd/pos/pos
  follow_inodes true
  refresh_interval 5s
  enable_stat_watcher false # To ensure that TailWathcer recognizes rotation
  rotate_wait 30s
  <parse>
    @type none
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Your Error Log

No error.

Additional context

We can set rotate_wait 0 for a workaround.