If Loki becomes unreachable Containers are stuck

[frzb]

It is a know upstream issue:

• https://grafana.com/docs/loki/latest/clients/docker-driver/#know-issues
• https://github.com/grafana/loki/issues/2361

The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the drive will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

Feeding the Docker container logs via Promtail or Fluentd into Loki and not over the Loki Docker logging driver should be sufficient as work-around.