cri-dockerd icon indicating copy to clipboard operation
cri-dockerd copied to clipboard

No workarounds for logging spam "docker does not support reopening container log files", when not using json-file logging driver

Open benileo opened this issue 3 months ago • 1 comments

Expected Behavior

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.509505   35502 container_log_manager.go:253] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.512587   35502 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

The SPAM is real.... for a 10 node cluster we had over 200,000 log lines of this nature in 1 hour. There are no "workarounds" for users who do not use the 'json-file' logging driver. Suggested workaround, from https://github.com/Mirantis/cri-dockerd/issues/35

fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.

Are users just subject to the pain? I saw in one thread that using a stub logging manager with cri-dockerd could work. Is there any documentation to help implement this?

This is our daemon configuration.

{
    "max-concurrent-downloads": 6,
    "exec-opts": [
        "native.cgroupdriver=systemd"
    ],
    "storage-driver": "overlay2",
    "live-restore": true,
    "log-driver": "fluentd",
    "log-opts": {
        ...
    }
}

Specifications

  • Version: Kubernetes 1.29.2
  • Platform: v0.3.11/cri-dockerd_0.3.11.3-0.ubuntu-jammy_amd64.deb

benileo avatar Mar 18 '24 22:03 benileo