Need the best way to fetch GW logs

[rahullepakshi]

Presently, I think the procedure we have to get gw logs to ACK few things or debug failures is by using journalctl. But the unit/service will not exist if GW exits/ crashes for some or the other reasons.

We need to have the best procedure to capture and the best place to store this while GW is running and even it exits/ crashes

[gbregman]

When I set

Storage=persistent

in /etc/systemd/journald.conf I can see the old log messages even after stopping the gateway and rebooting the host.

[caroav]

We should find out what other services in Ceph are doing to be able to have persistent logs.

[gbregman]

If we don't want to rely on journald, we can make sure to backup the json log file created by docker. We can find the path for this log file by running:

docker inspect -f '{{.LogPath}}' <CONTAINER-ID>

We can configure the usage of this log file by changing the content of /etc/docker/daemon.json . We can defined there the rotation policy and limit the size and number of log files.

[rahullepakshi]

Reopened this issue as when the GW crashes, logs are not available at all. More important is to analyse logs when GW crashes/ exits. We need logs at GW node at /var/log/ceph/ as like any other daemon below

[root@argo025 /]# ls /var/log/ceph/6d6c3c1a-c5ba-11ee-b2d3-ac1f6b0a1534
ceph-client.ceph-exporter.argo025.log              ceph-osd.3.log-20240209.gz  ceph-volume.log              ceph-volume.log-20240216.gz  ceph-volume.log-20240219.gz
ceph-client.ceph-exporter.argo025.log-20240209.gz  ceph-osd.9.log              ceph-volume.log-20240214.gz  ceph-volume.log-20240217.gz  ceph-volume.log-20240220.gz
ceph-osd.3.log                                     ceph-osd.9.log-20240209.gz  ceph-volume.log-20240215.gz  ceph-volume.log-20240218.gz

The same node has nvmeof GW
[root@argo025 /]# systemctl -a | grep nvmeof
  [email protected]_pool.argo025.wrpsgb.service
 loaded    active   running   Ceph nvmeof.nvmeof_pool.argo025.wrpsgb for 6d6c3c1a-c5ba-11ee-b2d3-ac1f6b0a1534