[warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s

[Joshua-Igoni]

Describe the bug

Deployed a set of containers, with docker-compose using journals as the logging driver, used fluentd to capture logs using the @systemd input plugin. apparently not getting any logs sent to elasticsearch or to file. config settings are pasted below for your perusal.

To Reproduce

Using fluentd as a container in dockercompose with @systemd as input plugin, specifying all the required parameters.

Expected behavior

tail the logs in /var/log/journal directory

Your Environment

- Fluentd version:v1.14-debian-1
- TD Agent version:
- Operating system:Debian
- Kernel version:

Your Configuration

<source>
  @type systemd
  tag dockerd
  path /var/log/journal
  read_from_head true
  fluentd_systemd_conf disable
</source>

<filter dockerd>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

<match dockerd>
  @type copy
  <store>
    @type file
    path /outputs/iotc.log
    compress gzip
    append true
    <buffer>
      @type memory
      flush_mode immediate
      flush_at_shutdown true
      flush_thread_count 6
    </buffer>
  </store>
  <store>
    @type elasticsearch
    host "<my_host_ip>"
    port 9200
    logstash_format true
    scheme https
    ssl_verify false
    user "<user>"
    password "<my_password>"
    <buffer>
      flush_at_shutdown true
      flush_interval 1s
    </buffer>
   </store>
</match>

Your Error Log

2022-09-18 09:02:08 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:09 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:10 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:11 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:12 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:13 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:14 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:15 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:16 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:17 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:18 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:19 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:20 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:21 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:22 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:23 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:24 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:25 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:26 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:27 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:28 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:29 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:30 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:31 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:32 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:33 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:34 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s
2022-09-18 09:02:35 +0000 [warn]: #0 Systemd::JournalError: No such file or directory retrying in 1s

Additional context

If this was running in a kubernettes cluster according to my own findings, tunning the "FLUENTD_SYSTEMD_CONF" to disable fixes this, but is there a way to do the same in this context?.

Someone should help me understand, Thanks in advance