[BUG]:容器日志文件收集，一个日志文件打开多个文件描述符

[caijinxu]

Describe the bug A clear and concise description of what the bug is, ideally within 20 words. 容器文件日志收集，一个日志文件打开多个文件描述符，最后触发以下报错： [2022-09-21 04:36:24.752141] [warning] [000033] /src/core/event_handler/EventHandler.cpp:582 too many open files:skip this read operation log path:/logtail_host/xxxx/warning-2022-09-20.log [2022-09-21 04:36:24.752146] [error] [000033] /src/core/reader/LogFileReader.cpp:1007 log file reader fd limit, too many open files:/logtail_host/xxxx/warning-2022-09-21.log : limit:25000 查看文件描述符打开数量 [root@ip-10-53-0-57 ilogtail]# ls -l /proc/11/fd |wc -l 25057 查看/proc/11/fd 内容，可以看到多个文件描述符指向通一个文件 lr-x------ 1 root root 64 Sep 21 01:46 17355 -> /logtail_host/xxxx/warning-2022-09-21.log lr-x------ 1 root root 64 Sep 21 01:46 17382 -> /logtail_host/xxxx/warning-2022-09-21.log lr-x------ 1 root root 64 Sep 21 01:46 17401 -> /logtail_host/xxxx/warning-2022-09-21.log .... 每次check container path update flag:true ，文件描述符使用就增长 [2022-09-21 04:42:45.520378] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2642 check container path update flag:true size:1 [2022-09-21 04:42:45.520394] [info] [000011] /src/core/observer/network/NetworkObserver.cpp:54 hold on:observer [2022-09-21 04:42:45.520408] [info] [000011] /src/core/controller/EventDispatcherBase.cpp:1137 main thread:start update config [2022-09-21 04:42:45.525535] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:98 logtail plugin HoldOn:start [2022-09-21 04:42:45.529024] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:102 logtail plugin HoldOn:success cost:4 iLogtail Running Environment Please provide the following information:

• ilogtail version: 1.1.1

• Yaml configuration:

• ilogtail.LOG: [2022-09-21 04:36:24.752141] [warning] [000033] /src/core/event_handler/EventHandler.cpp:582 too many open files:skip this read operation log path:/logtail_host/xxxx/warning-2022-09-20.log [2022-09-21 04:36:24.752146] [error] [000033] /src/core/reader/LogFileReader.cpp:1007 log file reader fd limit, too many open files:/logtail_host/xxxx/warning-2022-09-21.log : limit:25000

• logtail_plugin.LOG:

[caijinxu]

ilogtail version: sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.1.1

[yyuuttaaoo]

ilogtail version: sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.1.1

你好，与此相关的logtail配置文件和应用yaml配置能否脱敏后贴一下？我这里比较关注采集路径的配置，透明时请保留层级信息(如有2层就写a/b/warning.log)。另外问一下是什么容器环境呢？

[caijinxu]

ilogtail version: sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:1.1.1

你好，与此相关的logtail配置文件和应用yaml配置能否脱敏后贴一下？我这里比较关注采集路径的配置，透明时请保留层级信息(如有2层就写a/b/warning.log)。另外问一下是什么容器环境呢？

日志文件：/a/b/c/d/e/warn.log

logtail配置

enable: true
inputs:
- Type: file_log
  LogPath: /a/b/c/d
  FilePattern: '*.log'
  ContainerFile: true
  MaxDepth: 3
  ContainerInfo:
    K8sContainerRegex: aa
    K8sNamespaceRegex: bb
    ExternalEnvTag:
      _node_ip_: _node_ip_
      _node_name_: _node_name_
processors:
- Type: processor_split_log_regex
  SplitRegex: \d+\.\d+\.\d+\s\d+:\d+:\d+.*
  SplitKey: content
  PreserveOthers: true
flushers:
- Type: flusher_kafka
  Brokers:
  - 192.168.1.43:9092
  Topic: abc-logs

容器中的这个目录通过hostPath挂载到主机上，容器内和主机上目录结构一样

        - hostPath:
            path: /a/b/c/d/
            type: DirectoryOrCreate
          name: logpath

容器环境： ACK v1.20.11-aliyun.1 Docker 19.3.5

[yyuuttaaoo]

这个问题我们会持续跟进

[hanzoi]

同样的问题，看起来光一个pod同一个日志文件就有很多句柄

[yyuuttaaoo]

问题已经定位，会尽快发版修复