Add RFC-5424 syslog forwarding support to `journald`

[cipriancraciun]

Component

systemd-journald

Is your feature request related to a problem? Please describe

At the moment, when configuring ForwardToSyslog = yes, systemd-journald uses the UNIX socket domain /run/systemd/journal/syslog to forward messages using RFC-3164.

However, as #19251 describes (concerning the ingestion of syslog messages), sometimes it is useful to also have the forwarded messages conforming to RFC-5424 (instead of the current RFC-3164).

Describe the solution you'd like

Add a new /etc/systemd/journald.conf configuration option that allows the administrator to specify which protocol should the forwarded messages follow.

For example:

ForwardToSyslogProtocol = RFC5424
ForwardToSyslogProtocol = RFC3164

---

I don't think the proposed solution in #19251 of using xattr to mark the supported protocol directly on the socket is a workable solution here, mainly because all currently existing syslog servers don't actually support setting xattr's on listening sockets.

Describe alternatives you've considered

No response

The systemd version you checked that didn't have the feature you are asking for

255

[poettering]

Frankly, I am not convinced we should really beef up the forward-to-syslog stuff, because it has the major issue that there's no buffering, i.e. you can never catch up, anything listening will not be able to catch up with what was logged before it started listening.

The various syslog implementations I am aware of nowadays have plugins to read the data they need directly from the journal, and that does provide the buffering so that they can catch up with what was logged while they were offline.

I think rather then beefing this concept up we probably should deprecate it instead, i.e. keep it around and working as it is now, but drop it from the docs, and hide it in the default .conf file we install.

[cipriancraciun]

Indeed, the syslog protocol and implementations has all the issues you've mentioned (and perhaps more).

However, it is also a reasonably simple protocol to implement, and thus there are many small (non-enterprise-ish) tools out there that do implement it (and for which it wouldn't be feasible to implement systemd support).

Thus, having support for it (both in the RFC-3164 and RFC-5424 variants) could offer an easy escape hatch for those that need it, without too much overhead on either side.

---

For example, in my particular use-case, I have written myself a small Go-based syslog server that just stores logged messages in archived (xz / zstd) files. (I know it's not 100% reliable and it might miss messages due to UDP packets drops or in the time it is down, but that is fine.) I use it primarily for HAProxy request logs, but now I have added MQTT support to it for live-watching-filtering of logs when I need them. So, I wanted to also feed in the system log, where I have already configured everything to properly log to journald.

I could for example try to support the journald natively (either via the export format, or reading the log files), however that has two major downsides:

• most implementations for Go rely on the C library that systemd provides; thus linking to them would mean I no longer get easy cross-compilation from Go native build tools; (I now get statically linked binaries;)
• adding support for journald adds lots of complexity (as compared to syslog that I already support); (thus for the moment I'll just patch the code to support the old RFC-3164 format for this socket;)

What I describe here might be viewed as a "me" problem, which I can easily work-around. However this is just an use-case for supporting syslog (and the RFC-5424 format). There might be other similar use-cases out there.

---

I think rather then beefing this concept up we probably should deprecate it instead, i.e. keep it around and working as it is now, but drop it from the docs, and hide it in the default .conf file we install.

I think that would lead to even more perceived lock-in from the systemd ecosystem... (Which I think we already have enough...)

[cipriancraciun]

I've tried to approach this problem from multiple angles, and I fail to find a solution that works:

• apparently, journald although it states that it supports RFC-3164, when it writes the log line, it fails to also add the hostname;

  • thus the hostname information is not conveyed, and if one doesn't translate the UDP source IP to recover the hostname, all that information is lost forever; (and it wouldn't even work with machines behind NAT;)
  • granted, technically it is still compliant with RFC-3164, but by this measure, any random sequence of characters is valid according to RFC-3164;

• there seems to be journalctl -f -o short (or -o short-precise) that displays almost a RFC-3164 syslog line, except it fails to add the <pri> tag at the beginning of the line, thus forwarding this output to a syslog server would not be interpreted as proper RFC-3164;

• there is a small experimental project https://github.com/systemd/systemd-netlogd that should support reading the native journald files and forwarding everything to the target syslog server; it also supports RFC-5424, thus it might be an alternative; however:

  • it is not built by default on many Linux distributions (like OpenSUSE);
  • for some reason it has some bug (https://github.com/systemd/systemd-netlogd/issues/102) that just combines multiple messages into a single UDP packet;

Thus at the moment, even if one doesn't want to rely on RFC-5424, systemd-journald syslog forwarding doesn't work when forwarding these messages to a non-local syslog server.