conmon icon indicating copy to clipboard operation
conmon copied to clipboard

Max log size not applied when container restarts

Open pascapascapascapasca opened this issue 3 years ago • 0 comments

The max log size option is set to 200MB through log_size_max = 209715200 in /etc/containers/containers.conf (no other option is uncommented in it), but the container logs exceed 200MB. This seems to happen when a container restarts and thus the conmon process is restarted. In my case, podman restarts the container every few minutes and the bytes written by the previous conmon process do not seem to be taken in count when applying the max log size.

Versions CentOS 7

conmon --version conmon version 2.0.21 commit: efe288263ebb1bab4673d2cfbe1a352f3c7d27c1-dirty

runc --version runc version 1.0.0-rc10 commit: 4791ca3d31174576ddd8f4ff562283339196efa9 spec: 1.0.1-dev

rpm -qa | grep containers containers-common-1.2.0-3.el7.x86_64

podman version Version: 2.1.1 API Version: 2.0.0 Go Version: go1.15.2 Built: Wed Oct 28 07:37:02 2020 OS/Arch: linux/amd64

Results obtained This container's log file is over 800MB: ls -lah /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log -rw-------. 1 root root 839M Jun 1 14:05 /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log

I can see that the --log-size-max option is applied with the right value by conmon. ps -ef | grep conmon | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log root 23953 1 0 14:04 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -u 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata -p /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/pidfile -n fluent-sd-pod-fluent-sd --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log --log-size-max 209715200 --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7

Container has been restarted recently: date Tue Jun 1 14:36:06 CDT 2021

ps -eo pid,lstart,cmd | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log 17386 Tue Jun 1 14:32:21 2021 <...>

Results expected Container log file size not to exceed 200MB.

pascapascapascapasca avatar Jun 01 '21 19:06 pascapascapascapasca