beats
beats copied to clipboard
Use journalctl to read Journald logs
We need to replace the current github.com/coreos/go-systemd/v22/sdjournal
we use to read Journald logs by calling journalctl
direct.
The problem with github.com/coreos/go-systemd/v22/sdjournal
is that it relies on libsystemd and it is crashing with SIGBUS in multiple Linux distros, aside from that there are also other errors while reading the journal, the related issues:
- https://github.com/elastic/beats/issues/34077
- https://github.com/elastic/beats/issues/32782
- https://github.com/elastic/beats/issues/39352
The OpenTelemetry collector contrib calls journalctl
directly and does not experience the same SIGBUS crash as Filebaeat. Example of OpenTelemetry calling journalctl
: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/4322a3826e618666e679b5257d65441ed013ddf1/pkg/stanza/operator/input/journald/input.go#L67
I did some testing calling journalctl
directly to filter messages and it could get all messages without any problems. OpenTelemetry was mostly successful but it had a couple of crashes (the input stopped working, not the whole binary) related to the sending queue getting full and the same "bad message" crash we see very often on Filebeat (this is easy to reproduce on Debian 11).
Because it will require Filebeat to use the fork
syscall, ideally, we should only enable this syscall if the journald input is being used.
The focus of this issue is to get the core journald input working with journalctl
and remove all the CGO dependency brought by `github.com/coreos/go-systemd/v22/sdjournal.
Filtering options like units
, syslog_identifiers
, transports
, include_matches
, etc can be done as part of follow up tasks.