Azure service container logs need to be rotated

[pekajota]

Expected behavior

Should be able to access Manager/Workers host file system in order to clean-up log files

Actual behavior

As ssh sessions are directed to the agent container, there is no way to access host file system directories other than the ones that are automatically mounted.

Information

I have almost all file system taken but can't find who is using it: swarm-manager000000: df Filesystem 1K-blocks Used Available Use% Mounted on overlay 30831524 28023996 1218332 96% / tmpfs 7168368 4 7168364 0% /dev tmpfs 7168368 0 7168368 0% /sys/fs/cgroup tmpfs 7168368 165104 7003264 2% /etc /dev/sda1 30831524 28023996 1218332 96% /home tmpfs 7168368 165104 7003264 2% /mnt shm 7168368 0 7168368 0% /dev/shm /dev/sda1 30831524 28023996 1218332 96% /etc/ssh tmpfs 7168368 165104 7003264 2% /lib/modules tmpfs 7168368 165104 7003264 2% /lib/firmware /dev/sda1 30831524 28023996 1218332 96% /var/log /dev/sda1 30831524 28023996 1218332 96% /etc/hosts /dev/sda1 30831524 28023996 1218332 96% /etc/hostname /dev/sda1 30831524 28023996 1218332 96% /etc/resolv.conf tmpfs 1433676 1816 1431860 0% /var/run/docker.sock /dev/sda1 30831524 28023996 1218332 96% /var/lib/waagent tmpfs 7168368 165104 7003264 2% /usr/local/bin/docker /dev/sdb1 209713148 121824 209591324 0% /mnt/resource Output of du: swarm-manager000000: sudo du / -h -d 1 1.5M /sbin 0 /proc 111.6M /usr 1.2M /etc 7.0M /lib 16.0K /media 4.0K /srv 8.0K /tmp 4.0K /dev 12.0K /run 172.0K /root 0 /sys 720.0K /home 4.0K /mnt 88.0M /var 1.9M /bin 32.0K /opt 8.0K /daemons 7.5M /WALinuxAgent 219.6M /

[ddebroy]

@pekajota you can access host file system using something like the following: docker run --rm -it --privileged --pid=host justincormack/nsenter1 /bin/ash

You may want to check if the waagent's log files are taking up a lot of space. We fixes a log rotation issue with the log file recently. Which version are you on?

[pekajota]

Hi @ddebroy, thanks for the info. I can now access the host.

I've just found out that the source of 16GB worth of logs is docker4x/l4controller-azure:17.06.0-ce-azure2. It is printing out logs in debug mode. Do you know what might have caused this?

[pekajota]

@ddebroy I just found out that dockerd runs with debug enabled by default:

dockerd --pidfile=/run/docker.pid -H unix:///var/run/docker.sock --debug --storage-driver overlay2

Is this an expected behavior?

[ddebroy]

@pekajota thanks for reporting this issue. It seems the l4controller container log (generated by the l4controller that keeps the Azure load balancer and Swarm ports in sync) is not being rotated. We will enable this and is a bug in our init logic.

[ddebroy]

Changed title to reflect the issue