logstash-input-file icon indicating copy to clipboard operation
logstash-input-file copied to clipboard

Fixed issue of re-reading rotated files

Open punxism opened this issue 4 years ago • 2 comments
trafficstars

This is a fix for https://github.com/logstash-plugins/logstash-input-file/issues/238

Premis

  1. logstash checks the file uniqueness using the inode number, so even if sincedb exists, it does not fully support changing the file name in all cases.
  2. In a cloud environment, app rebooting occurs frequently due to deployment using docker, and as a result, tens of gigabytes or peta logs are read again.

Solution

  1. If the path of sincedb is changed and the inode is the same, it is recognized as a different file and the file is read again.
  2. However, even when file rotation occurs, it is always read again, which caused a problem. To solve the problem, look at the path of sincedb and start with this path and if the inode is not changed, it is regarded as a rotated file and the value of sincedb is maintained.

punxism avatar Sep 17 '21 01:09 punxism

❌ Author of the following commits did not sign a Contributor Agreement: a439c2159668d09d3bc0338e05158049bdd14e7b

Please, read and sign the above mentioned agreement if you want to contribute to this project

❌ Author of the following commits did not sign a Contributor Agreement: 5040fab

Please, read and sign the above mentioned agreement if you want to contribute to this project

Done.

punxism avatar Sep 23 '21 01:09 punxism