journald: Corrupted journal logs due to early backup on shutdown/reboot

[Kartoffelbauer]

Hi there, I'm a big fan of log2ram and use it everywhere. However, I've encountered some very annoying problems with journald in conjunction with log2ram.

The Problem

I've noticed that on every shutdown or reboot, journald always marks the last journal logs in /var/log/journal/<machine id>/ as corrupted (with a ~ at the end). This is persistent and there isn't a single archived file without the tilde (~) at the end.

The reason for this seems to be that log2ram creates the backup (/var/log/hdd.log) before the journald.service exits, and therefore the journal log included in the backup is incomplete. So when log2ram loads the backup into memory at the next boot and journald looks for the latest logs, it will determine that they are incomplete and therefore mark them as corrupt (by adding a tilde ~).

This in turn creates a whole new problem, as systemd in its current version on Raspberry Pi OS does not delete these corrupted logs and therefore ignores the setting for SystemMaxUse. This causes the journal folder in /var/log to grow and grow until it fills all the memory allocated by log2ram.

Possible solution

I'm no expert on this, but what actually needs to happen is that log2ram creates the backup after journald.service has finished writing to the journal logs. For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Version info

OS: Raspberry Pi OS bullseye (latest) log2ram: v1.6.1

[nodefeet]

For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Sorry for hijacking your important issue, but wouldn't that basically bypass log2ram?

[azlux]

Hi, Can you try to modify the systemd service with systemctl edit log2ram.service, this create an override file here /etc/systemd/system/log2ram.service.d/override.conf (you can delete this file once you've finish testing.

Can you try adding more service into the Before= instruction ? Especially the systemd-journal-flush.service how is one you're showing making issue in your case.

[Kartoffelbauer]

For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Sorry for hijacking your important issue, but wouldn't that basically bypass log2ram?

Yes, in this case journald will not write anything to /var/log anymore. Instead, everything is logged to /run/log (which is also mounted in RAM) and therefore log2ram is bypassed entirely.

[Kartoffelbauer]

Hi, Can you try to modify the systemd service with systemctl edit log2ram.service, this create an override file here /etc/systemd/system/log2ram.service.d/override.conf (you can delete this file once you've finish testing.

Can you try adding more service into the Before= instruction ? Especially the systemd-journal-flush.service how is one you're showing making issue in your case.

I will try that in the comming days. Thank you very much.

UPDATE: Today I tried your suggestion, but unfortunately nothing changed. Problem remains the same.

[solracsf]

Try this: https://unix.stackexchange.com/a/609594/527701 It may help you analyze where it breaks and how to solve it.

[HyP3r-]

I have created for that a pull request: https://github.com/azlux/log2ram/pull/200

[Kartoffelbauer]

I have created for that a pull request: #200

Thank you very much! journald now works properly with log2ram.