CrashLoopBackOff for fluentd pods

[tedam]

I have deployed kebrnetes kluster without efk enabled: kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready master,node 5d v1.9.2+coreos.0 node2 Ready master,node 5d v1.9.2+coreos.0 node3 Ready node 5d v1.9.2+coreos.0 node4 Ready node 5d v1.9.2+coreos.0

Then, I installed efk (kubectl apply -f logging) Problem with ES I solved as described here But still have problem with fluentd pods. They have status "CrashLoopBackOff", "Running" or "Completed" depending on I don't know what: [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 4 2m fluentd-qt9r7 0/1 CrashLoopBackOff 4 2m fluentd-wfp56 0/1 CrashLoopBackOff 4 2m fluentd-wj8wg 0/1 CrashLoopBackOff 3 1m [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 4 2m fluentd-qt9r7 0/1 CrashLoopBackOff 4 2m fluentd-wfp56 0/1 CrashLoopBackOff 4 2m fluentd-wj8wg 0/1 CrashLoopBackOff 3 1m [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 4 2m fluentd-qt9r7 0/1 CrashLoopBackOff 4 2m fluentd-wfp56 0/1 CrashLoopBackOff 4 2m fluentd-wj8wg 0/1 Completed 4 1m [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 4 2m fluentd-qt9r7 0/1 CrashLoopBackOff 4 2m fluentd-wfp56 0/1 Completed 5 3m fluentd-wj8wg 0/1 Completed 4 1m [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 4 3m fluentd-qt9r7 0/1 CrashLoopBackOff 4 2m fluentd-wfp56 0/1 Completed 5 3m fluentd-wj8wg 0/1 CrashLoopBackOff 4 2m [root@tdmkube-kube-master logging]# kubectl get pods -n logging|grep fluentd fluentd-dwx7q 0/1 CrashLoopBackOff 5 3m fluentd-qt9r7 0/1 CrashLoopBackOff 5 3m fluentd-wfp56 0/1 CrashLoopBackOff 5 4m fluentd-wj8wg 0/1 CrashLoopBackOff 4 2m

They restarted continiusly, and change status. kubectl logs fluentd-wj8wg shows nothing (empty output)

[root@tdmkube-kube-master logging]# kubectl describe pod fluentd-wj8wg -n logging shows this: Name: fluentd-wj8wg Namespace: logging Node: node2/10.28.79.148 Start Time: Wed, 07 Feb 2018 15:46:08 +0300 Labels: app=fluentd controller-revision-hash=1676977040 pod-template-generation=1 Annotations: Status: Running IP: 10.233.75.36 Controlled By: DaemonSet/fluentd Containers: fluentd: Container ID: docker://5788e95d64c8f02f5ebfca9d69d27c357e79b44c1fb03757cf89042f63ddd51f Image: gcr.io/google_containers/fluentd-elasticsearch:1.20 Image ID: docker-pullable://gcr.io/google_containers/fluentd-elasticsearch@sha256:de60e9048b3b79c4e53e66dc3b8ee58b4be2a88395a4b33407f8bbadc9548dfb Port: Command: /bin/sh -c /usr/sbin/td-agent -vv 2>&1 >>/var/log/fluentd.log State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 07 Feb 2018 15:52:14 +0300 Finished: Wed, 07 Feb 2018 15:52:16 +0300 Ready: False Restart Count: 6 Limits: memory: 200Mi Requests: cpu: 100m memory: 200Mi Environment: Mounts: /etc/td-agent from fluentd-conf (rw) /var/lib/docker/containers from varlibdockercontainers (ro) /var/log from varlog (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-j2sfr (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: varlog: Type: HostPath (bare host directory volume) Path: /var/log HostPathType: varlibdockercontainers: Type: HostPath (bare host directory volume) Path: /var/lib/docker/containers HostPathType: fluentd-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: fluentd-conf Optional: false default-token-j2sfr: Type: Secret (a volume populated by a Secret) SecretName: default-token-j2sfr Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/unreachable:NoExecute Events: Type Reason Age From Message

---

Normal SuccessfulMountVolume 7m kubelet, node2 MountVolume.SetUp succeeded for volume "varlibdockercontainers" Normal SuccessfulMountVolume 7m kubelet, node2 MountVolume.SetUp succeeded for volume "varlog" Normal SuccessfulMountVolume 7m kubelet, node2 MountVolume.SetUp succeeded for volume "fluentd-conf" Normal SuccessfulMountVolume 7m kubelet, node2 MountVolume.SetUp succeeded for volume "default-token-j2sfr" Normal Created 6m (x4 over 7m) kubelet, node2 Created container Normal Started 6m (x4 over 7m) kubelet, node2 Started container Normal Pulled 5m (x5 over 7m) kubelet, node2 Container image "gcr.io/google_containers/fluentd-elasticsearch:1.20" already present on machine Warning BackOff 1m (x24 over 7m) kubelet, node2 Back-off restarting failed container

Whats is problem here?

[rudymccomb]

Did you ever figure out this issue? @tedam

[jerem1664]

Hi, I encounter exactly the same issue. Did someone solved it ?

[jerem1664]

I find this in the log file /var/log/fluentd.log :

2018-05-09 15:56:00 +0000 [error]: fluent/supervisor.rb:369:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: pods is forbidden: User "system:serviceaccount:logging:default" cannot list pods at the cluster scope"

I manage to start fluentd daemonset by creating an serviceAccount for efk and adding :

serviceAccountName: efk

to fluentd-daemonset.yaml

I hope it will help.