cri-dockerd
cri-dockerd copied to clipboard
No workarounds for logging spam "docker does not support reopening container log files", when not using json-file logging driver
Expected Behavior
Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.509505 35502 container_log_manager.go:253] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files"
Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.512587 35502 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files"
The SPAM is real.... for a 10 node cluster we had over 200,000 log lines of this nature in 1 hour. There are no "workarounds" for users who do not use the 'json-file' logging driver. Suggested workaround, from https://github.com/Mirantis/cri-dockerd/issues/35
fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.
Are users just subject to the pain? I saw in one thread that using a stub logging manager with cri-dockerd could work. Is there any documentation to help implement this?
This is our daemon configuration.
{
"max-concurrent-downloads": 6,
"exec-opts": [
"native.cgroupdriver=systemd"
],
"storage-driver": "overlay2",
"live-restore": true,
"log-driver": "fluentd",
"log-opts": {
...
}
}
Specifications
- Version: Kubernetes 1.29.2
- Platform: v0.3.11/cri-dockerd_0.3.11.3-0.ubuntu-jammy_amd64.deb