journald: multiple journal cursor tracking

[jpds]

A note for the community

No response

Problem

I have some host hypervisors which use virtiofsd to expose the journald within the VMs to the host systems.

When I run journalctl -m on the host, I can see the log entries from the VM systems merged into this view.

However, vector running on the host does not appear to pick up on these log entries as the VM has a very different cursor record. I can see on the host:

vector      6654  0.1  0.0 282752 129240 ?       Ssl  14:37   0:01 /nix/store/7yidmj1jxyj2xhfl75lgcxbagincaarc-vector-0.46.1/bin/vector --config /nix/store/02vsrs3525x845f81frmarbwqxqbaak5-validate-vector-conf
vector      6739  0.0  0.0  60032  8872 ?        S    14:37   0:00  \_ journalctl --follow --all --show-cursor --output=json --after-cursor=s=f36492e269e54d59afbe1d19bdab0240;i=17c2;b=1293f27ab2114de5b68c3c552b350f4c;m=364078e7;t=636d40ed78de4;x=bac589a3c0576f58 --merge

Here are two log entries next to each other which have almost the same __REALTIME_TIMESTAMP, but different __CURSOR:

{"__SEQNUM":"3549","_CAP_EFFECTIVE":"0","_HOSTNAME":"vm","PRIORITY":"6","_EXE":"/nix/store/k2ghjdwsjsyr9fmhi92yfj3xg0kz5r1c-erlang-27.3.4/lib/erlang/erts-15.2.7/bin/beam.smp","_BOOT_ID":"395d67139c3a4ace8335545451651018","SYSLOG_IDENTIFIER":"servicectl","_RUNTIME_SCOPE":"system","_TRANSPORT":"stdout","__REALTIME_TIMESTAMP":"1749135671585868","_SYSTEMD_SLICE":"system.slice","_STREAM_ID":"35fc15de82e6438687192be7661beeba","_SYSTEMD_INVOCATION_ID":"faab3bcc65da4283b1f6d967f1310153","_CMDLINE":"...,"_SYSTEMD_UNIT":"service.service","_PID":"1186","SYSLOG_FACILITY":"3","MESSAGE":"2025-06-05 15:01:11.585595+00:00 ...","__SEQNUM_ID":"5d0f6f94737e4c3c8859303f69ffe05c","_UID":"219","_SYSTEMD_CGROUP":"/system.slice/service.service","_COMM":"beam.smp","_MACHINE_ID":"cf4b21792bb46fd87171c12d6841a2ce","__MONOTONIC_TIMESTAMP":"2291515945","__CURSOR":"s=5d0f6f94737e4c3c8859303f69ffe05c;i=ddd;b=395d67139c3a4ace8335545451651018;m=8895c229;t=636d4625e6c4c;x=91b8ad8cbac8a12c","_GID":"219"}
{"MESSAGE":"...","__SEQNUM_ID":"f36492e269e54d59afbe1d19bdab0240","_BOOT_ID":"1293f27ab2114de5b68c3c552b350f4c","__REALTIME_TIMESTAMP":"1749135671762328","__CURSOR":"s=f36492e269e54d59afbe1d19bdab0240;i=1f74;b=1293f27ab2114de5b68c3c552b350f4c;m=89ca089b;t=636d462611d98;x=f8fdab69a8cc038","_SOURCE_BOOTTIME_TIMESTAMP":"2312253810","_SOURCE_MONOTONIC_TIMESTAMP":"2312253810","PRIORITY":"4","__MONOTONIC_TIMESTAMP":"2311719067","SYSLOG_IDENTIFIER":"kernel","_TRANSPORT":"kernel","__SEQNUM":"8052","SYSLOG_FACILITY":"0","_RUNTIME_SCOPE":"system","_MACHINE_ID":"...","_HOSTNAME":"host"}

Within the journald directory, the files are organized like this:

jdavies@host /v/l/journal> eza -lR --tree
drwxr-sr-x@    - root  5 Jun 14:22 .
drwxr-sr-x@    - root  5 Jun 14:19 ├── 80b9fd6cc10d43809b54540290af43f4
.rw-r-----@ 8.4M root  5 Jun 14:21 │   └── system.journal
drwxr-sr-x@    - root  5 Jun 14:24 ├── ced76ade963d44ca812c622d43b89367
.rw-r-----@  17M root  5 Jun 15:19 │   ├── system.journal
.rw-r-----@ 8.4M root  5 Jun 15:08 │   └── user-1000.journal
lrwxrwxrwx     - root  5 Jun 14:22 └── cf4b21792bb46fd87171c12d6841a2ce -> /var/lib/vm/service/journal/cf4b21792bb46fd87171c12d6841a2ce
jdavies@host /v/l/journal> eza -lR --tree /var/lib/vm/service/journal/cf4b21792bb46fd87171c12d6841a2ce
drwxr-sr-x@    - root  5 Jun 14:27 /var/lib/vm/service/journal/cf4b21792bb46fd87171c12d6841a2ce
.rw-r-----@ 8.4M root  5 Jun 15:19 ├── system.journal
.rw-r-----@ 8.4M root  5 Jun 14:39 └── user-1000.journal

Perhaps vector should scan this directory for *.journal files and track a different cursor for each?

Configuration

[sources.journald]
current_boot_only = false
extra_args = ["--merge"]
type = "journald"

Version

0.46.1

Debug Output

Example Data

No response

Additional Context

No response

References

No response

[jpds]

OK, I just realized that instead of having a single journald source recurse through a directory structure looking for *.journal fles... I could just define multiple journald sources with different journal_directory options set.

This might be a worthwhile enhancement though?

[pront]

OK, I just realized that instead of having a single journald source recurse through a directory structure looking for *.journal fles... I could just define multiple journald sources with different journal_directory options set.

This might be a worthwhile enhancement though?

Hi @jpds, that seems like a good enhancement. Today, journald has no built-in mechanism to glob or recurse through subdirectories.