Disconnect between logrotate and emonhub rotation of log files

[borpin]

I'm sure this has come up before. There is a disconnect/conflict between the self rotation of logs by emonhub and the logrotate schemes. There is also inconsistency in log file creation & permissions depending on where it is done.

https://github.com/openenergymonitor/emonhub/blob/1215a9f930057a7049d4f2985ce35a195e02c595/src/emonhub.py#L321-L323

https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/etc/logrotate.d/emonhub

Add to this the service files and logging and the install scripts for emonhub and EmonScripts and emonhub logging is all a bit messed up.

https://github.com/openenergymonitor/emonhub/blob/4539b9e928e3f7b5bc3dd1922440cf7e725ac6b9/install.sh#L103-L110

https://github.com/openenergymonitor/emonhub/blob/4539b9e928e3f7b5bc3dd1922440cf7e725ac6b9/service/emonhub.service#L8-L16

There was some discussion here https://github.com/openenergymonitor/emonpi/issues/103

Where the internal python rotation happens, the subsequent rotated log files will not be rotated by logrotate. You will end up with a discontinuous log file (which would not help debugging).

As set currently, if the logfile happens to be between 3M and 5M in size when logrotate runs, the file will be rotated and only in this circumstance.

I think there should be a systematic review of the logging for emonhub.

My suggestion

If there is a no log file specified in the emonhub.service file, emonhub logging reverts to stderr, which ends up in rsyslog.log.

This logging can then be directed to a log file using a file /etc/rsyslog.d/10-emonhub.conf. [edit] - updated to latest format.

# https://www.rsyslog.com/doc/v8-stable/configuration/templates.html

template(name="EmonhubMsg" type="list") {
    property(name="hostname")
    constant(value=" ")
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg" droplastlf="on" )
    constant(value="\n")
    }

if ( $programname == 'emonhub' or $programname == 'emonhub.py' ) then {
    action(type="omfile" file="/var/log/emonhub/emonhub.log" template="EmonhubMsg" FileCreateMode="0644" dirCreateMode="0744")
    stop
}

This would then be correctly rotated by logrotate.

However, with the current logrotate settings mean that logrotate only runs daily, but the rotation settings mean the log is rotated weekly (unless too big).

Moving to hourly rotation can be solved by a logrotate drop-in to change the timer to hourly with minimal impact on resources and then rotate once the size is greater than 5M.

/lib/systemd/system/logrotate.timer.d/00_emonhub.conf

[Timer]
OnCalendar=
OnCalendar=hourly

Finally, the logrotate config needs amending slightly. Assuming logrotate is run hourly, the file will rotate when greater than 5M (but no earlier), but at least 3M, keep 14 files, compress on next rotation.

/var/log/emonhub/emonhub.log {
    size 5M
    minsize 3M
    maxsize 5M
    rotate 14
    compress
    delaycompress
    olddir /var/log.old/emonhub
    createolddir 775 root root
    renamecopy
}

Service file would therefore be simplified as well.

[Unit]
Description=emonHub data multiplexer
# The config file lives in /etc/emonhub/emonhub.conf
# The log file lives in /var/log/emonhub/emonhub.log
Requires=var-log.mount
After=var-log.mount network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf
User=pi

Restart=always
RestartSec=5

SyslogIdentifier=emonhub

[Install]
WantedBy=multi-user.target

Currently running this on a NOV22 EmonPi.

** Install and update scripts need considering ** Probably needs removing.

https://github.com/openenergymonitor/emonhub/blob/4539b9e928e3f7b5bc3dd1922440cf7e725ac6b9/install.sh#L103-L115

[borpin]

Related https://github.com/openenergymonitor/EmonScripts/issues/172

[borpin]

My only issue right now is that the emonhub log viewer in emoncms thinks there isn't a log file.

[MyForest]

I don't want to de-rail this effort, but it does look as though this is where comments should be made.

I do think you should do what works for you.

I wanted to say that I haven't stored logs "on-host" for about seven years now, I ship them off-host to another place so I can see them even when hosts go away or if they hang / crash. I know that isn't how most people view their emonhub instances so please feel free to ignore this approach unless you felt it could be useful. Specifically, it means I no longer have these log rotation problems, but of course I do have a lot of other problems caused by the alternate approach.

[borpin]

I do think you should do what works for you.

It is more about what works for 95% of users, really and to be more robust for them when running on Pis.

I wanted to say that I haven't stored logs "on-host" for about seven years now, I ship them off-host to another place so I can see them even when hosts go away or if they hang / crash.

This shouldn't make any difference to you as the log file is still generated just in a different way. On update you would pick up the modified service file and the installation of the rsyslog configuration.

Do you use log2ram to shift them? Are you running on a Pi or 'proper' hardware? TBH, I simply store the logs on the LXC host and rotate them.

[MyForest]

Are you running on a Pi or 'proper' hardware?

I don't want to de-rail things too much, so I'll briefly describe my setup. Mostly it'll be a lesson in what to avoid because it's too complicated.

I have EmonCMS plus all my custom monitoring (e.g. pywws, currentcost) running in containers, but I'm not running emonhub (which is why I'm trying to shoosh myself here). Those containers are hosted in Kubernetes which itself is hosted in a container running on regular PC (because it's one of twenty workloads.) I run a logging daemonset in the Kubernetes cluster which takes all the logs and posts them to a third-party logging system which handles things like querying, retention and structured logging. I don't hold on to any logs locally at all, everything logs to STDOUT / STDERR and the logging agent does the transport.

In some instances, such as mine, there are logs created on the host but they are ephemeral and so no rotation is required, they just get deleted when the containers cycle. In other scenarios STDOUT / STDERR are hooked directly to the logging system in-process so you don't even have any logs being stored (but you are vulnerable to losing the last logs still in the buffer when a process crashes).

[borpin]

I don't want to de-rail things too much, so I'll briefly describe my setup. Mostly it'll be a lesson in what to avoid because it's too complicated.

This is primarily aimed at emonhub users and Pis. If you do neither, then these changes will not affect you :)

Personally, I run a full emoncms setup in an LXC on PVE. Totally reliable and very easy to setup and maintain and automatic snapshots every night. What's not to like :)