[Agent][Filebeat][Log] Symbolic links are not follow correctly in some cases

[kuisathaverat]

For confirmed bugs, please report:

• Version: 7.13.0-SNAPSHOT
• Operating System: Linux CentOS 8.3 (Kubernetes 1.18.16-gke.502 and 1.19.8-gke.1600)
• Steps to Reproduce:

• create a GKE cluster
• deploy a few pods that generate logs (certmanager, ingress, and a Jenkins instance in my case)
• deploy the Elastic Agent with the configuration at https://github.com/elastic/beats/tree/master/deploy/kubernetes/elastic-agent-standalone
• the logs will show that there is a filebeat process restarting continuously

We configured an Elastic Agent standalone to grab logs and metrics on a k8s cluster, we see on logs that filebeat is restarting continuously. None of the logs point to the real issue nor the configuration that is causing the issue, in order to troubleshoot the issue the only way is to disable one by one all the inputs until finding the culprit. To have the configuration name in logs or in the document on Elasticsearch could help to make the troubleshooting easy.

We found the culprit input that causes the issue:

      - name: container-log
        type: logfile
        use_output: default
        meta:
          package:
            name: log
            version: 0.4.6
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: generic
            symlinks: true
            paths:
              - /var/log/containers/*${kubernetes.container.id}.log

After playing with a few settings I have tried to point the input to the real files, this made the trick, and not the logs are reported and filebeat is not restarting continuously. I have seen that the original configuration point to a symbolic link of a symbolic link, I think is related.

      - name: container-log
        type: logfile
        use_output: default
        meta:
          package:
            name: log
            version: 0.4.6
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: generic
            symlinks: true
            paths:
              - /var/lib/docker/containers/*/*-json.log
              # - /var/log/pods/*${kubernetes.container.id}.log/*/*.log
              # - /var/log/containers/*${kubernetes.container.id}.log

Elastic Agent logs

2021-04-29T15:42:39.616Z	INFO	log/reporter.go:40	2021-04-29T15:42:39Z - message: Application: filebeat--7.13.0-SNAPSHOT[24fc74a8-9997-4147-91b7-cce7f7442379]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-04-29T15:42:39.725Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-04-29T15:42:39.725Z	ERROR	log/reporter.go:36	2021-04-29T15:42:39Z - message: Application: filebeat--7.13.0-SNAPSHOT[24fc74a8-9997-4147-91b7-cce7f7442379]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-04-29T15:42:39.728Z	INFO	status/reporter.go:236	Elastic Agent status changed to: 'online'
2021-04-29T15:42:39.728Z	INFO	log/reporter.go:40	2021-04-29T15:42:39Z - message: Application: filebeat--7.13.0-SNAPSHOT[24fc74a8-9997-4147-91b7-cce7f7442379]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-04-29T15:42:39.728Z	INFO	log/reporter.go:40	2021-04-29T15:42:39Z - message: Application: filebeat--7.13.0-SNAPSHOT[24fc74a8-9997-4147-91b7-cce7f7442379]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-04-29T15:42:48.702Z	INFO	application/periodic.go:99	No configuration change
2021-04-29T15:42:58.703Z	INFO	application/periodic.go:99	No configuration change
2021-04-29T15:42:59.768Z	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-04-29T15:42:59.768Z	ERROR	log/reporter.go:36	2021-04-29T15:42:59Z - message: Application: filebeat--7.13.0-SNAPSHOT[24fc74a8-9997-4147-91b7-cce7f7442379]: State changed to FAILED: config blocks unsuccessfully generated: required 'object', but found 'string' in field 'app' - type: 'ERROR' - sub_type: 'FAILED'

Filebeta logs

[elastic_agent.filebeat][info] Home path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64] Config path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64] Data path: [/usr/share/elastic-agent/state/data/run/default/filebeat--7.13.0-SNAPSHOT] Logs path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64/logs]

....

[elastic_agent.filebeat][error] kubernetes: Querying for pod failed with error: pods "gke-apm-ci-k8s-cluster-pool-2-e8852348-50qt" not found

...

[elastic_agent.filebeat][error] kubernetes: Querying for pod failed with error: pods "gke-apm-ci-k8s-cluster-pool-2-e8852348-50qt" not found

...

[elastic_agent.filebeat][error] config blocks unsuccessfully generated: required 'object', but found 'string' in field 'app'
[elastic_agent.filebeat][info] Status change to Failed: config blocks unsuccessfully generated: required 'object', but found 'string' in field 'app'
[elastic_agent.filebeat][error] config blocks unsuccessfully generated: required 'object', but found 'string' in field 'app'
[elastic_agent.filebeat][info] Status change to Failed: config blocks unsuccessfully generated: required 'object', but found 'string' in field 'app'

[elastic_agent.filebeat][info] Home path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64] Config path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64] Data path: [/usr/share/elastic-agent/state/data/run/default/filebeat--7.13.0-SNAPSHOT] Logs path: [/usr/share/elastic-agent/state/data/install/filebeat-7.13.0-SNAPSHOT-linux-x86_64/logs]

[elasticmachine]

Pinging @elastic/obs-dc (Team:Obs-DC)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[artemlive]

I have the same issue. So any ideas on how to deal with this? If we use the path as /var/lib/docker/containers/*/*-json.log we won't be able to filter logs by the k8s metadata fields like 'kubernetes.pod.name', namespace, etc.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ph]

@jlind23 this seems something we should prioritize.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!