systemd doesn't read logs from persistent storage by default

[orgads]

Bug Report

Describe the bug If journald logs are stored as volatile (/run/log/journal), systemd input plugin reads the logs correctly.

But if persistent storage is used (/var/log/journal), no logs are read.

To Reproduce

• Steps to reproduce the problem: Use the following configuration on a machine that has the logs stored in /var/log/journal.

[INPUT]
    Name              systemd
    Tag               systemd
    Lowercase         On
    Strip_Underscores On
    DB                /fluent-bit/db/cursor.db

[OUTPUT]
    Name   stdout
    Match  *

Expected behavior Logs should be read from /var/log/journal if it exists.

Your Environment

• Version used: 3.0.2
• Configuration: Above
• Environment name and version (e.g. Kubernetes? What version?): Docker with /var/log and /run/log mounted as volumes.
• Operating System and version: Debian 12, Redhat 8.
• Filters and plugins: systemd

Additional context I deploy on various systems, some have the logs in run and some in var, so I cannot use Path.

In https://github.com/fluent/fluent-bit/blob/master/plugins/in_systemd/systemd_config.c#L103, sd_journal_open(&ctx->j, SD_JOURNAL_LOCAL_ONLY); is used to open the journal. I ran strace and found that it does open /var/log/journal and gets a file descriptor for it, but it doesn't search inside it...

I tried to configure 2 entries with Path - one for each path, but since each of them doesn't always exist, fluent-bit failed to start.

[patrick-stephens]

Not sure I follow the failing to start with two separate path entries, I'd expect it just to report it can't read one - same as if it did not have access for example. Unless it's to do with nesting or something strange.

Are you using two different tail inputs or specifying two paths in a single input?

[orgads]

I tried this:

[INPUT]
    Name              systemd
    Tag               systemd
    Systemd_Filter    _COMM=systemd
    Path              /run/log/journal
    Lowercase         On
    Strip_Underscores On
    DB                /fluent-bit/db/cursor1.db

[INPUT]
    Name              systemd
    Tag               systemd
    Systemd_Filter    _COMM=systemd
    Path              /var/log/journal
    Lowercase         On
    Strip_Underscores On
    DB                /fluent-bit/db/cursor2.db

and it failed with this error:

[2024/04/16 16:45:11] [ info] [input:systemd:systemd.0] initializing
[2024/04/16 16:45:11] [ info] [input:systemd:systemd.0] storage_strategy='memory' (memory only)
[2024/04/16 16:45:11] [error] [/src/fluent-bit/plugins/in_systemd/systemd_config.c:81 errno=2] No such file or directory
[2024/04/16 16:45:11] [error] [input:systemd:systemd.0] given path /run/log/journal is invalid
[2024/04/16 16:45:11] [error] [input:systemd:systemd.0] cannot initialize
[2024/04/16 16:45:11] [error] failed initialize input systemd.0
[2024/04/16 16:45:11] [error] [engine] input initialization failed

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[orgads]

@patrick-stephens ping