fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

Update in_tail to use storage plugins instead of pos_file

Open tagomoris opened this issue 8 years ago • 1 comments

in_tail is using a file to record the positions of files, to restart reading from there after Fluentd process restarts. But pos_file is formatted in custom binary, and hard to read/edit. And it requires any fixed path to be configured.

Storage plugins (especially storage_local) can store any data on paths configured via root_dir. It's very easy to manage, and make configuration files smaller, by reducing each paths (including pos_file) from plugin configuration sections. in_tail should be updated to use storage plugins for positions.

Points to be considered:

  • How to convert pos_file to storage plugin data? online update? any converter scripts?
  • How to manage/clean-up old data? #1126

tagomoris avatar Jan 04 '17 05:01 tagomoris

How do we get this feature (defect) fixed? This is creating a situation for us - as the POS file is growing rather quickly and the process has to continue to comb thru this file. We see constant CPU cycles even after no additional data has been written to the tailed log.

mrkstate avatar Jul 06 '18 12:07 mrkstate