Logs not being collected unless `logsConfigContainerCollectAll` enabled

[goodgravy]

Describe what happened: We followed these instructions to install the operator in a k8s 1.22 cluster. We found that the logs from our application pods were not collected by the agents to be sent to DataDog unless agent.log.logsConfigContainerCollectAll was set to true.

However, in this case, we also get a ton of noise in the logs from ancillary pods.

Describe what you expected: We used (an older) version of the operator in a k8s 1.20 cluster, and saw that the logs from identical app pods were collected by agents – even though agent.log.logsConfigContainerCollectAll is false in this previous cluster.

We were expecting for the agents in the new cluster to collect log output from our application pods in the same was as happened on the original cluster.

Steps to reproduce the issue: Not sure, sorry :(

Our application pods do have two containers running in them – the app itself and a proxy. Our SO post contains as much info as we have.

Additional environment details (Operating System, Cloud provider, etc):

• EKS 1.22 (and 1.20 on the original system which showed different behaviour)
• The datadog-operator helm release version is 0.8.6 in the new cluster (and 0.7.5 on the original cluster)
• The app version of the helm release is 0.8.1 (and 0.7.2 on the original cluster)

[goodgravy]

Right now, I have enabled logsConfigContainerCollectAll in the DatadogAgent resource, then have a gory exclusion filter in our DataDog log configuration. The goal of this exclusion filter is to not have our logs cluttered with stuff from istio-proxy, prometheus, datadog-agent – all that kind of thing.

[jeffreyflynt]

Right now, I have enabled logsConfigContainerCollectAll in the DatadogAgent resource, then have a gory exclusion filter in our DataDog log configuration. The goal of this exclusion filter is to not have our logs cluttered with stuff from istio-proxy, prometheus, datadog-agent – all that kind of thing.

Can you share a sample exclusion snippet?

[levan-m]

Hello, sorry for not getting to the issue on time. Operator v0.x is no longer supported and we recommend migrating to most recent version of v1.x.

Please open a new issue if there is anything blocking migration or you experience same issue in v1.x.