fluentd
fluentd copied to clipboard
Update in_tail to use storage plugins instead of pos_file
in_tail
is using a file to record the positions of files, to restart reading from there after Fluentd process restarts.
But pos_file is formatted in custom binary, and hard to read/edit. And it requires any fixed path to be configured.
Storage plugins (especially storage_local) can store any data on paths configured via root_dir
. It's very easy to manage, and make configuration files smaller, by reducing each paths (including pos_file) from plugin configuration sections.
in_tail
should be updated to use storage plugins for positions.
Points to be considered:
- How to convert pos_file to storage plugin data? online update? any converter scripts?
- How to manage/clean-up old data? #1126
How do we get this feature (defect) fixed? This is creating a situation for us - as the POS file is growing rather quickly and the process has to continue to comb thru this file. We see constant CPU cycles even after no additional data has been written to the tailed log.