严重bug log-pilot 不释放inode

[jedy-tian]

log-pilot收集storm worker log 配置如下

• type: log enabled: true paths: - /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-*.log scan_frequency: 10s fields_under_root: true fields: index: storm_094_worker topic: storm_094_worker docker_container: storm-0.9.4-supervisor tail_files: false close_inactive: 2h close_eof: false close_removed: true
  clean_removed: true close_renamed: false

日志目录列表如下 /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm # ls access.log worker-6800.log.6 worker-6801.log.6 worker-6803.log.5 worker-6804.log.5 worker-6805.log.5 worker-6806.log.7 worker-6807.log.7 worker-6808.log.7 worker-6809.log.7 logviewer.log worker-6800.log.7 worker-6801.log.7 worker-6803.log.6 worker-6804.log.6 worker-6805.log.6 worker-6806.log.8 worker-6807.log.8 worker-6808.log.8 worker-6809.log.8 metrics.log worker-6800.log.8 worker-6801.log.8 worker-6803.log.7 worker-6804.log.7 worker-6805.log.7 worker-6806.log.9 worker-6807.log.9 worker-6808.log.9 worker-6809.log.9 supervisor.log worker-6800.log.9 worker-6801.log.9 worker-6803.log.8 worker-6804.log.8 worker-6806.log worker-6807.log worker-6808.log worker-6809.log worker-6800.log worker-6801.log worker-6802.log worker-6803.log.9 worker-6804.log.9 worker-6806.log.1 worker-6807.log.1 worker-6808.log.1 worker-6809.log.1 worker-6800.log.1 worker-6801.log.1 worker-6803.log worker-6804.log worker-6805.log worker-6806.log.2 worker-6807.log.2 worker-6808.log.2 worker-6809.log.2 worker-6800.log.2 worker-6801.log.2 worker-6803.log.1 worker-6804.log.1 worker-6805.log.1 worker-6806.log.3 worker-6807.log.3 worker-6808.log.3 worker-6809.log.3 worker-6800.log.3 worker-6801.log.3 worker-6803.log.2 worker-6804.log.2 worker-6805.log.2 worker-6806.log.4 worker-6807.log.4 worker-6808.log.4 worker-6809.log.4 worker-6800.log.4 worker-6801.log.4 worker-6803.log.3 worker-6804.log.3 worker-6805.log.3 worker-6806.log.5 worker-6807.log.5 worker-6808.log.5 worker-6809.log.5 worker-6800.log.5 worker-6801.log.5 worker-6803.log.4 worker-6804.log.4 worker-6805.log.4 worker-6806.log.6 worker-6807.log.6 worker-6808.log.6 worker-6809.log.6 /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm #

但最终work*.log.9没有被释放出来 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) 29 /usr/bin/filebeat /host/data_sdd/cloud/storm-0.9.4-supervisor/data/logs/storm/worker-6807.log.9 (deleted) ~ # lsof |grep deleted|wc -l 16960 ~ # lsof |grep deleted| grep log.9 |wc -l 16960 ~ #

[chenquanzhao]

@jedy-tian 若某个日志文件采集完，并且无新日志再写入到本日志文件，一般filebeat会自动释放该文件句柄

[chen-tm]

我今天测试时间也遇到了，有解决方案吗

[silenceper]

同问，用filebeat 的模式跑了几天，大量被删除的文件句柄还保持，占用磁盘空间，导致node 上业务pod被驱逐