Logfile /var/log/icinga2/icinga2.log to stdout

[trickert76]

Hi, I'm running the image for a long time and the logfile of /var/log/icinga2/icinga2.log is getting bigger and bigger making the volume bigger and bigger.

I thought it is good practice to send all logs inside a container to stdout. How could I do that here - would it be possible to change the image that it links /var/log/icinga2/icinga2.log to /dev/stdout?

[trickert76]

Something like this in the Dockerfile RUN ln -sf /dev/stdout /var/log/icinga2/icinga2.log

[bebehei]

It's a simple icinga2 feature disable mainlog && supervisorctl restart icinga2 in your container. AFAIK, icinga2 always logs also onto stdout. PR welcome.

[trickert76]

Ok, I would make a branch with the RUN.. example - but are you sure about "icinga2 always logs also onto stdout"? Because I've logs at docker which are 2 days old and icinga2.log message only minutes ago. So there is a difference in stdout logging of Icinga and the icinga2.log

[jjethwa]

Thanks @trickert76 and @bebehei

[jjethwa]

@trickert76 I had to comment out as the icinga2 service would not start

 ln -sf /dev/stdout /var/log/icinga2/icinga2.log

[bebehei]

So, what was the point of the whole PR?

[jjethwa]

I'd need to do more troubleshooting to figure out why that line caused an issue. I had to spend time getting the 2.11.0 changes working properly instead

[trickert76]

Sorry for the delay. I was able to reproduce this today. I only check the "up" state of the docker and that isn't enough here (this is an issue for a healthcheck). Also when I make the ln in the running container and restart it, icinga2 backend failes to start without telling why - and the stdout log isn't used at all. I didn't check that, but I think, there are some file ops, that can't be used with links and the backend isn't aware of it.

Ok, so forget the PR. I can't find a way to configure Icinga to use a different location for the /var/log/icinga2/icinga2.log(is there somewhere a configuration option?). Alternatively mount the directory /var/log/icinga2 to avoid a growing icinga2 container (some Gigabytes per day).

[jjethwa]

Hi @trickert76

I would definitely recommend using a volume for /var/log/icinga2 so you can persist them over container restarts or upgrades.

[magnusbaeck]

With the default configuration of Docker Engine the container logs are never rotated so I don't see how getting Icinga to log to stdout solves any problem. It does however introduce a requirement that log shippers must be able to tail Docker container logs. If we're to have this behavior I suggest it's configurable via one of the entrypoint scripts instead.

My preferred way of dealing with log rotation would be to install logrotate in the image (unless it's already in place after the Buster upgrade; I haven't checked) and invoke the program on the Icinga and Apache logrotate configuration files (and possibly others) via the host's crontab. This method keeps the size of all logs down to a reasonable level.

[jjethwa]

Hi @magnusbaeck

I'll check around and see what some of the official recommendations from Docker, Kubernetes, etc say. Previously stdout and management via a volume was preferred. I'll do some research and get back to you. Thanks 😄

[magnusbaeck]

I'm sure the general recommendation is (still) to log to stdout, but that breaks down for fat images like this one.

[jjethwa]

@magnusbaeck

True and this is not the typical microservice type container 😉

[AlvaroBrey]

I was about to open an issue because logrotate is missing from the image, and found this issue instead.

In short, at least in image jordan/icinga2:2.10.3 which is what I'm using, both cron and logrotate are missing. This means the log rotation configs that are provided in /etc/logrotate.d/icinga2 are never executed and the logfile keeps growing indefinitely.

Just installing logrotate should be enough to fix this.

[jjethwa]

Hi @alvarobrey

Thanks for your findings. Would you like to submit a PR to add logrotate?