Promtail unable to handle log rotations on Kubernetes

[hussamsh]

Describe the bug I have loki and promtail running on my local -minikube- Kubernetes cluster, I also have a pod that generates a decent amount of logs per second. After a couple of hours of operation loki doesn't seem to receive any new logs from promtail even though my pod is healthy and generating logs consistently.

Restarting promtail or my pod resolves this issue and I can see new logs once again but after a couple of hours the logs stop again.

I suspect this is an issue with how promtail handles log rotation in Kubernetes

To Reproduce Steps to reproduce the behavior:

1. Start minikube [latest version]
2. Install loki stack using helm - I was also successful in replicating this with grafana cloud Kubernetes integrations.
3. Start a pod that generates enough logs to trigger a log rotation from Kubernetes
4. Observe that loki doesn't receive any new logs from promtail. Promtail stops collecting logs from that pod.

Expected behavior Promtail should be able to handle log rotations without dropping subsequent logs

Additional data --When switching promtail with fluent-bit collector I don't see this issue anymore which tells me that this is an issue with promtail.

Environment:

• Infrastructure: [kubernetes]
• Deployment tool: [helm]

Screenshots, Promtail config, or terminal output N/A

[chaudum]

Afaik, minikube uses systemd for logging (aka journald log driver for Docker) and therefore the built-in log rotation mechanism.

I did a brief search how journald rotates files, but could not find a satisfying answer. However, you may find some help in this document about Promtail and log rotation.

[Saisreenivas]

Facing same issue in our case as well. using kubelet(and does in-built log rotation).

According to the above document, there shouldn't be any data loss.

But if you see the image below, promtail has a lot of data loss.

[r0ka]

I have the same issue with K8s managed by EKS. Is there any news?