lumberjack
lumberjack copied to clipboard
natefinch
Fluentd cannot detected log being rotating
As title, Is it possible integrate with fluentd in-tail?
Fluentd stop updating the pos-file when rotation happened.
I see the same issue with simply 'tail'. If I run 'tail -F logfile' and the file gets rotated, tail doesn't detect the change (even though the '-F' flag should guarantee it).
I'm not sure how tail and fluentd work in this case. I can look into it. There may be some file descriptor magic logrotate etc do to keep tail on the right track.
To add to this, I was able to solve the issue for tail by using 'tail -F --max-unchanged-stats=5 logfile', which reloads the file when it no longer changes.
Using only '-F', I noticed that tail kept looking at the moved file (according to what I saw in /proc/
Yeah, that makes sense. Tail would follow the file descriptor, even if the name changed. When lumberjack rotates, it just renames the old file, which would keep the same file descriptor. I would need to look at how to fix that. I could copy the file and truncate the original, but then we'd have that file's contents on disk twice, and that could mean you'd be using too much disk space (although I guess that's the same disk space you'd use once the original file fills up to capacity, too. Hmm.)
Yes yes, that's what & why I wrote in https://github.com/natefinch/lumberjack/issues/115#issuecomment-739835573
I could copy the file and truncate the original
the copyturncate option which like logrotate do is a bad idea,
- it's not safe.
- it will introduce a heavy load and causes a large performance downgrade in a high-qps server.
Many companies (include us) were encountered the 2nd issue.
A related issue https://github.com/logrotate/logrotate/issues/269 and https://github.com/logrotate/logrotate/pull/361