k8s-infra chart does not collect logs in Oracle OKE

[patrijua]

We have installed SigNoz Helm chart (0.12.1) and also k8s-infra chart as well. K8s-infra chart is with default values (Oracle OKE 1.23.4). Problem is that kubernetes pod logs are not visible in SigNoz. In otel agent logs I can see that it detects the file to watch in node:

2023-03-20T11:57:39.619Z info fileconsumer/file.go:171 Started watching file {"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/*********/configuration/0.log"}

I have verified that file contains all the logs for the container. There are no errors visible in agent logs, signoz-otel-collector logs or clickhouse logs, but the logs are not visible in SigNoz. signoz-k8s-infra-otel-agent-nz7rk.log

It's hard to debug since one can not exec into agent container to check can it access the files on node:

kubectl exec -i -t -n platform signoz-k8s-infra-otel-agent-nz7rk -c signoz-k8s-infra-otel-agent -- sh -c "clear; (bash || ash || sh)" ERRO[0000] exec failed: unable to start container process: exec: "sh": executable file not found in $PATH command terminated with exit code 255

In nodes /var/log/messages there's log entries related to this:

Apr 5 07:24:19 oke-cul2jmprnmq-nr6xzjb6w6q-su3ppd6zdfa-0 setroubleshoot[1714595]: SELinux is preventing otelcol-contrib from read access on the directory /var/log/pods.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that otelcol-contrib should be allowed read access on the pods directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'otelcol-contrib' --raw | audit2allow -M my-otelcolcontrib#012# semodule -X 300 -i my-otelcolcontrib.pp#012

[patrijua]

We overcame the issue by new SELinux policy, but maybe some notes should be included in repo that there is a "challenge" when SELinux is used.

[prashant-shahi]

@patrijua Thanks for sharing your experience. I will look into it.

[prashant-shahi]

@euclidesflores Would it be possible for you to verify logs collections as well when you test OKE?

[euclidesflores]

@prashant-shahi yes, leave it with me. I will take a look.