beats icon indicating copy to clipboard operation
beats copied to clipboard

Use journalctl to read Journald logs

Open belimawr opened this issue 8 months ago • 1 comments

We need to replace the current github.com/coreos/go-systemd/v22/sdjournal we use to read Journald logs by calling journalctl direct.

The problem with github.com/coreos/go-systemd/v22/sdjournal is that it relies on libsystemd and it is crashing with SIGBUS in multiple Linux distros, aside from that there are also other errors while reading the journal, the related issues:

  • https://github.com/elastic/beats/issues/34077
  • https://github.com/elastic/beats/issues/32782
  • https://github.com/elastic/beats/issues/39352

The OpenTelemetry collector contrib calls journalctl directly and does not experience the same SIGBUS crash as Filebaeat. Example of OpenTelemetry calling journalctl: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/4322a3826e618666e679b5257d65441ed013ddf1/pkg/stanza/operator/input/journald/input.go#L67

I did some testing calling journalctl directly to filter messages and it could get all messages without any problems. OpenTelemetry was mostly successful but it had a couple of crashes (the input stopped working, not the whole binary) related to the sending queue getting full and the same "bad message" crash we see very often on Filebeat (this is easy to reproduce on Debian 11).

Because it will require Filebeat to use the fork syscall, ideally, we should only enable this syscall if the journald input is being used.

The focus of this issue is to get the core journald input working with journalctl and remove all the CGO dependency brought by `github.com/coreos/go-systemd/v22/sdjournal.

Filtering options like units, syslog_identifiers, transports, include_matches, etc can be done as part of follow up tasks.

belimawr avatar Jun 06 '24 15:06 belimawr