conmon should support syslog and none log drivers

[haircommander]

It should also support more, but these log drivers should be fairly trivial to implement

[goochjj]

PR #172

[PavelSosin-320]

@haircommander Now conmon 2.1 tries to write messages to DEBU[0000] /usr/bin/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c d8db6db55eff0baf9291c7e88eb475c5571b283ce572984bad82b878f6dee7d7 -u d8db6db55eff0baf9291c7e88eb475c5571b283ce572984bad82b878f6dee7d7 -r /usr/bin/crun -b /containers/storage/btrfs-containers/d8db6db55eff0baf9291c7e88eb475c5571b283ce572984bad82b878f6dee7d7/userdata -p /containers/storage/btrfs-containers/d8db6db55eff0baf9291c7e88eb475c5571b283ce572984bad82b878f6dee7d7/userdata/pidfile -n happy_jang --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level debug --syslog --conmon-pidfile /containers/storage/btrfs-containers/d8db6db55eff0baf9291c7e88eb475c5571b283ce572984bad82b878f6dee7d7/userdata/conmon.pid --exit- ... But it doesn't work well Looking inside code I see that the log target determination is static when logger endpoint is dynamic, the daemon serving the endpoint can be started via socket activation. It can't be too simple, at least, something like stat endpoint is must. In the case of stat's failure the triggered stack dump is better then proceeding silently from the maintenance point of view.

[haircommander]

sorry @PavelSosin-320 it's not clear to me what you're looking for. this issue is describing where container logs are written, not logs of the conmon process. That is determined by the --syslog flag which is passed by podman or cri-o. Most of the knobs you're describing are iin the podman backend, not conmon (unless I'm misunderstanding)

[PavelSosin-320]

@haircommander I observe the Podman container start issue that appears from Podman version 3.4.4 on Fedora 35 workstation with BTRFS and GNome 4, naturally. At the same time Journal logs started to fill with corrupted log files. Although systemd-journad mechanism, tested using systemd-cat and journalctl --user works OK for the same rootless user any attempt to start previously created container results in OCI Access violation. The used images are taken from the standard repositories, like Ubuntu. The SELinux labelling of the container storage and runtime looks OK. It looks like the error comes from Conmon and related to the writing to the journal. I compare the code of run.c of systemd conmon.c and they look very different. It looks like conmon makes decision whether to write log to journal using sd_journal is made statically when journal endpoint is activated dynamically via systemd socket activation. Although Fedora uses journal as a primary logging mechanism the debug log of Podman contains record /usr/bin/conmon messages will be logged to syslog, The code also doesn't pipe OCI runtime output to the journald stream socket like cat. I afraid that journald driver doesn't fulfill all journald-specific requirements and it causes both Podman start error and log corruption. P.S. Journald related issues are numerous.

[PavelSosin-320]

@haircommander After full cleanup and rerunning of podman start I found that the issue is actually dup of #https://github.com/containers/conmon/issues/67. Starting from Fedora, up to 34 Fedora had no syslog driver but it could be installed. For Fedora 35, like in Ubuntu the journal is the only option, no syslog daemon and unit exist and, so no related error is shown in systemctl status. Podman also fooling conmon passing --syslog option. Since gnome-logs uses journald as backend the syslog is not usable in conjunction with Gnome desktop. Any static check or configuration can't produce consistent behavior - if journald unit fails conmon will fluctuate, logs application will not display Podman errors. Everything depends on log endpoint socket.