[Telemetry Operator] avoid log explosion problem by excluding fluent-bit logs from collection

[a-thaler]

Description If there are logs produced by fluent-bit itself per ingested log line, it might lead to a log explosion problem as for every tailed line a new line gets created which gets tailed again. That's why typically logs of the log collector will not be ingested.

Gardener is doing that by a simple exclude of the relevant pod, see https://github.com/gardener/gardener/blob/master/charts/seed-bootstrap/charts/fluent-bit/templates/_fluent-bit-config.tpl#L27

Alternative is to define a dedicated tag for the fluent bit logs so that they are excluded by default but can be treated, like done in fluentd https://docs.fluentd.org/deployment/logging#capture-fluentd-logs which means having a dedicated tail input defined only for these logs.

As the second option can be added at any time, we should start with option 1 and exclude these logs from the standard application pipeline.

Criterias

• application logs of fluentbit will not be part of the default application input

[github-actions[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.