Unable to collect the logs when the DOCKER ROOT DIR is not consistent with different nodes

[wmwm061061]

1、fluent-bit 采集容器标准输出日志时，挂载的宿主机目录为 /var/log/

apiVersion: v1
kind: Pod
metadata:
  name: fluent-bit-46b47
spec:
  containers:
    volumeMounts:
    - mountPath: /var/log/
      name: varlogs
      readOnly: true
  volumes:
  - hostPath:
      path: /var/log
      type: ""
    name: varlogs

fluent-bit Input tail 插件采集日志的目录为 /var/log/containers/*.log

[Input]
    Name    tail
    Path    /var/log/containers/*.log

/var/log/containers 目录中的文件实际上都是软链接，它指向的 /var/log/pods 目录，而 /var/log/pods/*/*.log 文件也都是软链接，最终指向的是 DOCKER_ROOT_DIR/containers/*/*.log

这个 DOCKER_ROOT_DIR/containers 目录也会挂载到 fluent-bit 容器中

apiVersion: v1
kind: Pod
metadata:
  name: fluent-bit-46b47
spec:
  containers:
    volumeMounts:
    - mountPath: /var/lib/docker/containers
      name: varlibcontainers
  volumes:
  - hostPath:
      path: /var/lib/docker/containers
      type: ""
    name: varlibcontainers

2、fluent-bit 容器中的挂载的 DOCKER_ROOT_DIR 目录是从 fluentbit-operator 容器所在的宿主机获取的

apiVersion: v1
kind: Pod
metadata:
  name: fluentbit-operator-78fc579f6-crrt5
spec:
  initContainers:
  - command:
    - /bin/sh
    - -c
    - set -ex; echo DOCKER_ROOT_DIR=$(docker info -f {{.DockerRootDir}}) > /fluentbit-operator/fluent-bit.env

3、当某个 worker 节点的 DOCKER_ROOT_DIR 与 fluentbit-operator 所在的宿主机不一致时，这个 worker 节点的容器日志将无法采集

例如：

• fluentbit-operator 所在的宿主机的 DOCKER_ROOT_DIR 为 /var/lib/docker
• 某个 worker 节点的 DOCKER_ROOT_DIR 为 /home/docker

此时这个 worker 节点 /var/log/pods/*/*.log 软链接文件指向的是 /home/docker/containers/*/*.log ，fluent-bit 容器中找不到 /home/docker/containers/*/*.log 文件

原因是这个 worker 节点对应的 fluent-bit 容器挂载的 DOCKER_ROOT_DIR/containers 是 fluentbit-operator 所在的宿主机获取到的 /var/lib/docker/containers，而不是 /home/docker/containers

4、有什么办法可以让 fluent-bit 容器挂载的 DOCKER_ROOT_DIR 为所在节点 dockerd 的 DOCKER_ROOT_DIR 吗？

[wenchajun]

Maybe you can modify the Input plugin. The input.spec.tail.path property specifies the log collection address

[wenchajun]

Also, you can modify fluentbit.spec.containerLogRealPath to modify Container log path. Here is code https://github.com/wenchajun/fluentbit-operator/blob/master/api/fluentbitoperator/v1alpha2/fluentbit_types.go#L36