charts icon indicating copy to clipboard operation
charts copied to clipboard

Published 20 hours ago • verified publisher icon truecharts

failed to tail file, stat failed

Open NickScarpitti opened this issue 2 years ago • 10 comments

App Name

loki

SCALE Version

22.02.3

App Version

2.7.1_9.0.3

Application Events

2022-12-16 19:04:59
Applied LoadBalancer DaemonSet ix-loki-new/svclb-loki-new
2022-12-16 19:14:48
LoadBalancer Ingress IP addresses updated: 192.168.1.45
2022-12-16 19:18:03
Started container lb-tcp-13100
2022-12-16 19:18:03
Created container lb-tcp-13100
2022-12-16 19:17:54
Container image "rancher/klipper-lb:v0.3.4" already present on machine
2022-12-16 19:17:54
Add eth0 [172.16.5.185/16] from ix-net
2022-12-16 19:17:30
Created pod: svclb-loki-new-htrgw
Successfully assigned ix-loki-new/svclb-loki-new-htrgw to ix-truenas
2022-12-16 19:17:30
LoadBalancer Ingress IP addresses updated:
2022-12-16 19:17:23
Stopping container lb-tcp-3101
2022-12-16 19:17:23
Deleted pod: svclb-loki-new-2v7rk
2022-12-16 19:17:22
Job completed
2022-12-16 19:17:16
Started container loki-new-promtail-manifests
2022-12-16 19:17:16
Created container loki-new-promtail-manifests
2022-12-16 19:17:08
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:17:08
Add eth0 [172.16.5.184/16] from ix-net
Successfully assigned ix-loki-new/loki-new-promtail-manifests-q7cfr to ix-truenas
2022-12-16 19:16:56
Created pod: loki-new-promtail-manifests-q7cfr
2022-12-16 19:16:56
Job completed
2022-12-16 19:16:51
Started container loki-new-manifests
2022-12-16 19:16:51
Created container loki-new-manifests
2022-12-16 19:16:44
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:16:44
Add eth0 [172.16.5.183/16] from ix-net
Successfully assigned ix-loki-new/loki-new-manifests-s48rm to ix-truenas
2022-12-16 19:16:37
Created pod: loki-new-manifests-s48rm
2022-12-16 19:14:47
Started container lb-tcp-3101
2022-12-16 19:14:46
Created container lb-tcp-3101
2022-12-16 19:14:42
Container image "rancher/klipper-lb:v0.3.4" already present on machine
2022-12-16 19:14:42
Add eth0 [172.16.5.182/16] from ix-net
Successfully assigned ix-loki-new/svclb-loki-new-2v7rk to ix-truenas
2022-12-16 19:14:30
Created pod: svclb-loki-new-2v7rk
2022-12-16 19:14:30
Deleted pod: svclb-loki-new-l6f52
2022-12-16 19:14:30
Job completed
2022-12-16 19:14:25
Started container loki-new-promtail-manifests
2022-12-16 19:14:24
Created container loki-new-promtail-manifests
2022-12-16 19:14:17
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:14:17
Add eth0 [172.16.5.181/16] from ix-net
Successfully assigned ix-loki-new/loki-new-promtail-manifests-2hfv4 to ix-truenas
2022-12-16 19:14:11
Created pod: loki-new-promtail-manifests-2hfv4
2022-12-16 19:14:11
Job completed
2022-12-16 19:14:06
Started container loki-new-manifests
2022-12-16 19:14:06
Created container loki-new-manifests
2022-12-16 19:14:01
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:14:01
Add eth0 [172.16.5.180/16] from ix-net
Successfully assigned ix-loki-new/loki-new-manifests-5b2bh to ix-truenas
2022-12-16 19:13:55
Created pod: loki-new-manifests-5b2bh
0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
2022-12-16 19:06:14
Started container loki-new
2022-12-16 19:06:14
Created container loki-new
2022-12-16 19:06:05
Successfully pulled image "tccr.io/truecharts/loki:2.7.1@sha256:a94fe45062e6dfefeb9b9363e266744a6b43449f47521f8676d755462be2d872" in 32.438383531s
2022-12-16 19:05:49
Started container loki-new-promtail
2022-12-16 19:05:47
Created container loki-new-promtail
2022-12-16 19:05:39
Successfully pulled image "ghcr.io/truecharts/promtail:2.7.1@sha256:94be035cd342c5d4a088394fd68e3bb3f1071eeb1ab20aeb1886b7f14d842cce" in 21.09561472s
2022-12-16 19:05:32
Pulling image "tccr.io/truecharts/loki:2.7.1@sha256:a94fe45062e6dfefeb9b9363e266744a6b43449f47521f8676d755462be2d872"
2022-12-16 19:05:31
Started container prepare
2022-12-16 19:05:30
Created container prepare
2022-12-16 19:05:20
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:05:20
Add eth0 [172.16.5.179/16] from ix-net
2022-12-16 19:05:18
Pulling image "ghcr.io/truecharts/promtail:2.7.1@sha256:94be035cd342c5d4a088394fd68e3bb3f1071eeb1ab20aeb1886b7f14d842cce"
2022-12-16 19:05:17
Started container prepare
2022-12-16 19:05:14
Created container prepare
Successfully assigned ix-loki-new/loki-new-0 to ix-truenas
2022-12-16 19:05:08
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:05:08
Add eth0 [172.16.5.178/16] from ix-net
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-12-16 19:05:06
Successfully provisioned volume pvc-c9a8c5ae-9e0c-456e-b35e-b0dadd979def
2022-12-16 19:05:00
External provisioner is provisioning volume for claim "ix-loki-new/data-loki-new-0"
0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
Successfully assigned ix-loki-new/loki-new-promtail-54465f6787-rml5p to ix-truenas
2022-12-16 19:05:00
Created pod: svclb-loki-new-l6f52
2022-12-16 19:05:00
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-12-16 19:04:59
Created pod: loki-new-promtail-54465f6787-rml5p
2022-12-16 19:04:59
Scaled up replica set loki-new-promtail-54465f6787 to 1
2022-12-16 19:04:59
create Pod loki-new-0 in StatefulSet loki-new successful
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-12-16 19:04:59
create Claim data-loki-new-0 Pod loki-new-0 in StatefulSet loki-new success
2022-12-16 19:04:59
MountVolume.SetUp failed for volume "kube-api-access-nq82r" : object "ix-loki-new"/"kube-root-ca.crt" not registered
2022-12-16 19:04:59
Job completed
2022-12-16 19:04:55
Started container loki-new-promtail-manifests
2022-12-16 19:04:55
Created container loki-new-promtail-manifests
2022-12-16 19:04:50
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:04:50
Add eth0 [172.16.5.177/16] from ix-net
Successfully assigned ix-loki-new/loki-new-promtail-manifests-8cbhz to ix-truenas
2022-12-16 19:04:44
Created pod: loki-new-promtail-manifests-8cbhz
2022-12-16 19:04:44
Job completed
2022-12-16 19:04:39
Started container loki-new-manifests
2022-12-16 19:04:39
Created container loki-new-manifests
2022-12-16 19:04:33
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-16 19:04:33
Add eth0 [172.16.5.176/16] from ix-net
Successfully assigned ix-loki-new/loki-new-manifests-ldbxz to ix-truenas
2022-12-16 19:04:27
Created pod: loki-new-manifests-ldbxz

Application Logs

loki-promtail logs:
2022-12-17T00:21:24.136780709Z level=error ts=2022-12-17T00:21:24.048781939Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/prometheus-operator_prometheus-operator-5c7445d877-64jgw_d624ac67-7feb-4f13-aeff-1727c834f669/prometheus-operator/0.log: no such file or directory" filename=/var/log/pods/prometheus-operator_prometheus-operator-5c7445d877-64jgw_d624ac67-7feb-4f13-aeff-1727c834f669/prometheus-operator/0.log
2022-12-17T00:21:24.136813315Z level=error ts=2022-12-17T00:21:24.048999567Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-pihole_svclb-pihole-dns-dfcz5_072fab12-0bb9-49c0-910f-df69f67eb1fc/lb-udp-53/0.log: no such file or directory" filename=/var/log/pods/ix-pihole_svclb-pihole-dns-dfcz5_072fab12-0bb9-49c0-910f-df69f67eb1fc/lb-udp-53/0.log
2022-12-17T00:21:24.136847258Z level=error ts=2022-12-17T00:21:24.049337117Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-loki-new_loki-new-0_7269ec5f-ed81-45ec-b842-efc16cac8a8b/loki-new/0.log: no such file or directory" filename=/var/log/pods/ix-loki-new_loki-new-0_7269ec5f-ed81-45ec-b842-efc16cac8a8b/loki-new/0.log
2022-12-17T00:21:24.136884736Z level=error ts=2022-12-17T00:21:24.049481192Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27851340-v6qwz_ca4ad278-5b4d-407f-add8-3c42ff3b9c37/clamav/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27851340-v6qwz_ca4ad278-5b4d-407f-add8-3c42ff3b9c37/clamav/0.log
2022-12-17T00:21:24.136912955Z level=error ts=2022-12-17T00:21:24.049540104Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-recycler_recycler-ix-chart-86c7c956d8-fdj9p_f8680ef1-f47b-4b8a-ba76-fc0e4e53de53/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-recycler_recycler-ix-chart-86c7c956d8-fdj9p_f8680ef1-f47b-4b8a-ba76-fc0e4e53de53/ix-chart/0.log
2022-12-17T00:21:24.136951689Z level=error ts=2022-12-17T00:21:24.05021982Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-video-watcher_video-watcher-ix-chart-667f446f-zmg67_ef36717f-04da-4579-b285-a572e61996b2/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-video-watcher_video-watcher-ix-chart-667f446f-zmg67_ef36717f-04da-4579-b285-a572e61996b2/ix-chart/0.log
2022-12-17T00:21:24.136983309Z level=error ts=2022-12-17T00:21:24.050378856Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-loki-new_loki-new-0_7269ec5f-ed81-45ec-b842-efc16cac8a8b/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-loki-new_loki-new-0_7269ec5f-ed81-45ec-b842-efc16cac8a8b/prepare/0.log
2022-12-17T00:21:24.137013377Z level=error ts=2022-12-17T00:21:24.050604629Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-grafana_svclb-grafana-8sfcv_93473594-81ac-4967-8c82-7a0a0c55fe43/lb-tcp-10038/0.log: no such file or directory" filename=/var/log/pods/ix-grafana_svclb-grafana-8sfcv_93473594-81ac-4967-8c82-7a0a0c55fe43/lb-tcp-10038/0.log
2022-12-17T00:21:24.137053421Z level=error ts=2022-12-17T00:21:24.050645771Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-sonarr_sonarr-cb4cc9f9b-8dsch_ac7c8fac-f2b0-419f-8850-bcf802e36516/sonarr/0.log: no such file or directory" filename=/var/log/pods/ix-sonarr_sonarr-cb4cc9f9b-8dsch_ac7c8fac-f2b0-419f-8850-bcf802e36516/sonarr/0.log
2022-12-17T00:21:24.137082062Z level=error ts=2022-12-17T00:21:24.051147271Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_svclb-clam-av-clamav-milter-jkqp9_71bdbf18-8c75-4c25-af42-28c6df99f9b8/lb-tcp-7357/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_svclb-clam-av-clamav-milter-jkqp9_71bdbf18-8c75-4c25-af42-28c6df99f9b8/lb-tcp-7357/0.log
2022-12-17T00:21:24.137117454Z level=error ts=2022-12-17T00:21:24.051311444Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-traefik_svclb-traefik-lclcd_bddd507e-cad9-4930-9931-9f0555a5c05e/lb-tcp-9000/0.log: no such file or directory" filename=/var/log/pods/ix-traefik_svclb-traefik-lclcd_bddd507e-cad9-4930-9931-9f0555a5c05e/lb-tcp-9000/0.log
2022-12-17T00:21:24.137152384Z level=error ts=2022-12-17T00:21:24.05136051Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-traefik_traefik-6dd9df99ff-vtds8_d5fb07b8-2bd1-48bf-9c19-7b1eaefc1812/traefik/0.log: no such file or directory" filename=/var/log/pods/ix-traefik_traefik-6dd9df99ff-vtds8_d5fb07b8-2bd1-48bf-9c19-7b1eaefc1812/traefik/0.log
2022-12-17T00:21:24.137181759Z level=error ts=2022-12-17T00:21:24.052040865Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-sonarr_sonarr-cb4cc9f9b-8dsch_ac7c8fac-f2b0-419f-8850-bcf802e36516/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-sonarr_sonarr-cb4cc9f9b-8dsch_ac7c8fac-f2b0-419f-8850-bcf802e36516/prepare/0.log
2022-12-17T00:21:24.137224152Z level=error ts=2022-12-17T00:21:24.052231181Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-loki-new_loki-new-promtail-54465f6787-rml5p_e2338fa3-5b64-4054-9167-ed271dda63f9/loki-new-promtail/0.log: no such file or directory" filename=/var/log/pods/ix-loki-new_loki-new-promtail-54465f6787-rml5p_e2338fa3-5b64-4054-9167-ed271dda63f9/loki-new-promtail/0.log
2022-12-17T00:21:24.137264905Z level=error ts=2022-12-17T00:21:24.052500011Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/openvpn/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/openvpn/0.log
2022-12-17T00:21:24.137309271Z level=error ts=2022-12-17T00:21:24.052502631Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-jackett_jackett-67fcb478b8-s5hml_15203cf4-0973-49c6-95af-040dfa22c579/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-jackett_jackett-67fcb478b8-s5hml_15203cf4-0973-49c6-95af-040dfa22c579/prepare/0.log
2022-12-17T00:21:24.137346911Z level=error ts=2022-12-17T00:21:24.052972739Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-music-watcher_music-watcher-ix-chart-7f6bdffd79-twlx4_d36bfc2c-c4a7-4a8a-8505-9424331b32ea/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-music-watcher_music-watcher-ix-chart-7f6bdffd79-twlx4_d36bfc2c-c4a7-4a8a-8505-9424331b32ea/ix-chart/0.log
2022-12-17T00:21:24.137380320Z level=error ts=2022-12-17T00:21:24.05318122Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/prepare/0.log
2022-12-17T00:21:24.137425656Z level=error ts=2022-12-17T00:21:24.05322587Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/qbittorrent/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_qbittorrent-7bbcd556dc-fvxcq_159b9085-4003-466c-a162-d8a70d117bb0/qbittorrent/0.log
2022-12-17T00:21:24.137456640Z level=error ts=2022-12-17T00:21:24.05389802Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_svclb-qbittorrent-torrentudp-x9lfz_4f5772eb-4f2c-44d2-b4c3-a2189ae78ce7/lb-udp-6881/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_svclb-qbittorrent-torrentudp-x9lfz_4f5772eb-4f2c-44d2-b4c3-a2189ae78ce7/lb-udp-6881/0.log
2022-12-17T00:21:24.137497627Z level=error ts=2022-12-17T00:21:24.054054156Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-nweb-pg-bouncer_nweb-pg-bouncer-ix-chart-689f6b56-8x9nr_78bfa5ac-d96c-4b51-a2d8-20e16bd9ceb6/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-nweb-pg-bouncer_nweb-pg-bouncer-ix-chart-689f6b56-8x9nr_78bfa5ac-d96c-4b51-a2d8-20e16bd9ceb6/ix-chart/0.log
2022-12-17T00:21:24.137535071Z level=error ts=2022-12-17T00:21:24.054228292Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-hades_hades-plex-7dd7d446d6-58jc6_f395c096-1c0d-4364-9f57-cc1b696a3e36/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-hades_hades-plex-7dd7d446d6-58jc6_f395c096-1c0d-4364-9f57-cc1b696a3e36/prepare/0.log
2022-12-17T00:21:24.137566386Z level=error ts=2022-12-17T00:21:24.054375521Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27847020-ktmbf_fe47fe39-9945-4dd2-8562-b3fcdef87832/clamav/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27847020-ktmbf_fe47fe39-9945-4dd2-8562-b3fcdef87832/clamav/0.log
2022-12-17T00:21:24.137610643Z level=error ts=2022-12-17T00:21:24.054758892Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-grafana_grafana-7467c45d7f-zszlw_9190e5c1-6469-4b7a-a897-a93fbbdf3d66/grafana/0.log: no such file or directory" filename=/var/log/pods/ix-grafana_grafana-7467c45d7f-zszlw_9190e5c1-6469-4b7a-a897-a93fbbdf3d66/grafana/0.log
2022-12-17T00:21:24.137640413Z level=error ts=2022-12-17T00:21:24.054997238Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-radarr_radarr-85c86b8c96-7zspz_17ce4082-8258-4ea7-b11a-8a41212e07d4/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-radarr_radarr-85c86b8c96-7zspz_17ce4082-8258-4ea7-b11a-8a41212e07d4/prepare/0.log
2022-12-17T00:21:24.137675646Z level=error ts=2022-12-17T00:21:24.055080268Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-readarr_svclb-readarr-fchkb_8f403620-adeb-4bae-8bfa-8f1a241aad1e/lb-tcp-8787/0.log: no such file or directory" filename=/var/log/pods/ix-readarr_svclb-readarr-fchkb_8f403620-adeb-4bae-8bfa-8f1a241aad1e/lb-tcp-8787/0.log
2022-12-17T00:21:24.137709582Z level=error ts=2022-12-17T00:21:24.055886273Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_svclb-prometheus-hfz4n_40be054b-9dc1-4d75-96f8-a84987158570/lb-tcp-10086/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_svclb-prometheus-hfz4n_40be054b-9dc1-4d75-96f8-a84987158570/lb-tcp-10086/0.log
2022-12-17T00:21:24.137738223Z level=error ts=2022-12-17T00:21:24.055914661Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_clam-av-clamav-6d84b8dfb5-426dt_04bfd291-3819-46ef-af18-8614e07ba253/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_clam-av-clamav-6d84b8dfb5-426dt_04bfd291-3819-46ef-af18-8614e07ba253/prepare/0.log
2022-12-17T00:21:24.137777853Z level=error ts=2022-12-17T00:21:24.056217697Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-grafana_grafana-7467c45d7f-zszlw_9190e5c1-6469-4b7a-a897-a93fbbdf3d66/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-grafana_grafana-7467c45d7f-zszlw_9190e5c1-6469-4b7a-a897-a93fbbdf3d66/prepare/0.log
2022-12-17T00:21:24.137810070Z level=error ts=2022-12-17T00:21:24.056607695Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-audio-bookshelf_audio-bookshelf-audiobookshelf-567ff998c7-jdgz8_0c18628b-082c-469c-80b7-870aac061832/audio-bookshelf-audiobookshelf/0.log: no such file or directory" filename=/var/log/pods/ix-audio-bookshelf_audio-bookshelf-audiobookshelf-567ff998c7-jdgz8_0c18628b-082c-469c-80b7-870aac061832/audio-bookshelf-audiobookshelf/0.log
2022-12-17T00:21:24.137854234Z level=error ts=2022-12-17T00:21:24.056829764Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-radarr_svclb-radarr-99cl8_f56f22ce-8a50-4f47-9bb4-8403f3414cad/lb-tcp-7878/0.log: no such file or directory" filename=/var/log/pods/ix-radarr_svclb-radarr-99cl8_f56f22ce-8a50-4f47-9bb4-8403f3414cad/lb-tcp-7878/0.log
2022-12-17T00:21:24.137888156Z level=error ts=2022-12-17T00:21:24.057011818Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_clam-av-clamav-6d84b8dfb5-426dt_04bfd291-3819-46ef-af18-8614e07ba253/clam-av-clamav/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_clam-av-clamav-6d84b8dfb5-426dt_04bfd291-3819-46ef-af18-8614e07ba253/clam-av-clamav/0.log
2022-12-17T00:21:24.137926228Z level=error ts=2022-12-17T00:21:24.057378068Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-lidarr_svclb-lidarr-4fxmc_bc5c6c52-df75-4e96-b947-d010bf75de1f/lb-tcp-8686/0.log: no such file or directory" filename=/var/log/pods/ix-lidarr_svclb-lidarr-4fxmc_bc5c6c52-df75-4e96-b947-d010bf75de1f/lb-tcp-8686/0.log
2022-12-17T00:21:24.137964854Z level=error ts=2022-12-17T00:21:24.05770578Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-tdarr_svclb-tdarr-comm-tchgb_194fc8bc-34e4-4750-867c-54387464ef30/lb-tcp-8266/0.log: no such file or directory" filename=/var/log/pods/ix-tdarr_svclb-tdarr-comm-tchgb_194fc8bc-34e4-4750-867c-54387464ef30/lb-tcp-8266/0.log
2022-12-17T00:21:24.137997592Z level=error ts=2022-12-17T00:21:24.057744564Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-loki_svclb-loki-z6qfz_f3576f84-a88f-4f41-b636-708ad675dbf4/lb-tcp-3100/0.log: no such file or directory" filename=/var/log/pods/ix-loki_svclb-loki-z6qfz_f3576f84-a88f-4f41-b636-708ad675dbf4/lb-tcp-3100/0.log
2022-12-17T00:21:24.138040514Z level=error ts=2022-12-17T00:21:24.058111266Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-tdarr_svclb-tdarr-cs8gm_1d337551-9081-43eb-89c5-479c8d8f99bd/lb-tcp-8265/0.log: no such file or directory" filename=/var/log/pods/ix-tdarr_svclb-tdarr-cs8gm_1d337551-9081-43eb-89c5-479c8d8f99bd/lb-tcp-8265/0.log
2022-12-17T00:21:24.138070141Z level=error ts=2022-12-17T00:21:24.058417922Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/config-reloader/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/config-reloader/0.log
2022-12-17T00:21:24.138101972Z level=error ts=2022-12-17T00:21:24.058666069Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/prometheus/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/prometheus/0.log
2022-12-17T00:21:24.138146341Z level=error ts=2022-12-17T00:21:24.058843917Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-nweb-app_nweb-app-ix-chart-5747c964d9-87glb_2f7a2acd-5bbe-4ca6-a041-266d44710f97/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-nweb-app_nweb-app-ix-chart-5747c964d9-87glb_2f7a2acd-5bbe-4ca6-a041-266d44710f97/ix-chart/0.log
2022-12-17T00:21:24.138190356Z level=error ts=2022-12-17T00:21:24.059015469Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/init-config-reloader/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_prometheus-prometheus-prometheus-0_fad06d5f-1e06-4ebf-b08b-353da4e22a85/init-config-reloader/0.log
2022-12-17T00:21:24.138236313Z level=error ts=2022-12-17T00:21:24.059524943Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_openebs-zfs-node-4pkqx_5588fd88-4879-4d4f-9cbe-079cc807239a/openebs-zfs-plugin/0.log: no such file or directory" filename=/var/log/pods/kube-system_openebs-zfs-node-4pkqx_5588fd88-4879-4d4f-9cbe-079cc807239a/openebs-zfs-plugin/0.log
2022-12-17T00:21:24.138279879Z level=error ts=2022-12-17T00:21:24.059629645Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-nweb-api_nweb-api-ix-chart-9c6db4678-zrfbc_2bcd3e9f-e125-4c01-ac27-048f1b78f5ff/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-nweb-api_nweb-api-ix-chart-9c6db4678-zrfbc_2bcd3e9f-e125-4c01-ac27-048f1b78f5ff/ix-chart/0.log
2022-12-17T00:21:24.138325203Z level=error ts=2022-12-17T00:21:24.060001667Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-radarr_radarr-85c86b8c96-7zspz_17ce4082-8258-4ea7-b11a-8a41212e07d4/radarr/0.log: no such file or directory" filename=/var/log/pods/ix-radarr_radarr-85c86b8c96-7zspz_17ce4082-8258-4ea7-b11a-8a41212e07d4/radarr/0.log
2022-12-17T00:21:24.138370827Z level=error ts=2022-12-17T00:21:24.060180723Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_nvidia-device-plugin-daemonset-5r29m_5c76d277-3a3e-4a0a-ba5d-273cc21602a9/nvidia-device-plugin-ctr/0.log: no such file or directory" filename=/var/log/pods/kube-system_nvidia-device-plugin-daemonset-5r29m_5c76d277-3a3e-4a0a-ba5d-273cc21602a9/nvidia-device-plugin-ctr/0.log
2022-12-17T00:21:24.138417571Z level=error ts=2022-12-17T00:21:24.060396237Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/cnpg-system_cnpg-controller-manager-854876b995-lbwl2_2e49e4f1-c818-4646-a5d9-fe2d0fba8560/manager/0.log: no such file or directory" filename=/var/log/pods/cnpg-system_cnpg-controller-manager-854876b995-lbwl2_2e49e4f1-c818-4646-a5d9-fe2d0fba8560/manager/0.log
2022-12-17T00:21:24.138466097Z level=error ts=2022-12-17T00:21:24.060497541Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-readarr_readarr-64b59f546f-xnj4v_0c58b4a4-512f-4c29-8036-186603ff1383/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-readarr_readarr-64b59f546f-xnj4v_0c58b4a4-512f-4c29-8036-186603ff1383/prepare/0.log
2022-12-17T00:21:24.138513297Z level=error ts=2022-12-17T00:21:24.060524454Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-traefik_traefik-6dd9df99ff-vtds8_d5fb07b8-2bd1-48bf-9c19-7b1eaefc1812/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-traefik_traefik-6dd9df99ff-vtds8_d5fb07b8-2bd1-48bf-9c19-7b1eaefc1812/prepare/0.log
2022-12-17T00:21:24.138561442Z level=error ts=2022-12-17T00:21:24.061020165Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/metallb-system_controller-7597dd4f7b-fth44_4900c89d-915f-40b7-aea6-f093d28fddaf/controller/0.log: no such file or directory" filename=/var/log/pods/metallb-system_controller-7597dd4f7b-fth44_4900c89d-915f-40b7-aea6-f093d28fddaf/controller/0.log
2022-12-17T00:21:24.138610959Z level=error ts=2022-12-17T00:21:24.061060468Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-traefik_svclb-traefik-tcp-zfk4t_1928dd1e-ea57-4d58-a4e6-cd48554ff7b8/lb-tcp-80/0.log: no such file or directory" filename=/var/log/pods/ix-traefik_svclb-traefik-tcp-zfk4t_1928dd1e-ea57-4d58-a4e6-cd48554ff7b8/lb-tcp-80/0.log
2022-12-17T00:21:24.138658531Z level=error ts=2022-12-17T00:21:24.061319152Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-readarr_readarr-64b59f546f-xnj4v_0c58b4a4-512f-4c29-8036-186603ff1383/readarr/0.log: no such file or directory" filename=/var/log/pods/ix-readarr_readarr-64b59f546f-xnj4v_0c58b4a4-512f-4c29-8036-186603ff1383/readarr/0.log
2022-12-17T00:21:24.138703732Z level=error ts=2022-12-17T00:21:24.06146559Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prowlarr_svclb-prowlarr-brn7w_8d478f70-21aa-4273-a170-7545eab23bfd/lb-tcp-9696/0.log: no such file or directory" filename=/var/log/pods/ix-prowlarr_svclb-prowlarr-brn7w_8d478f70-21aa-4273-a170-7545eab23bfd/lb-tcp-9696/0.log
2022-12-17T00:21:24.138749057Z level=error ts=2022-12-17T00:21:24.061664461Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_prometheus-node-exporter-khtw8_d00bd723-b871-4c8a-9481-5fac83464bba/node-exporter/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_prometheus-node-exporter-khtw8_d00bd723-b871-4c8a-9481-5fac83464bba/node-exporter/0.log
2022-12-17T00:21:24.138793904Z level=error ts=2022-12-17T00:21:24.061691132Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prometheus_prometheus-kube-state-metrics-55fb94d768-tgnkz_95d5879a-f46e-4d8f-82e9-bb53159af74b/kube-state-metrics/0.log: no such file or directory" filename=/var/log/pods/ix-prometheus_prometheus-kube-state-metrics-55fb94d768-tgnkz_95d5879a-f46e-4d8f-82e9-bb53159af74b/kube-state-metrics/0.log
2022-12-17T00:21:24.138841478Z level=error ts=2022-12-17T00:21:24.061832321Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-wire-guard-easy_svclb-wire-guard-easy-wg-easy-64xjl_1a6f0098-d7c4-4be6-afd5-4fe4b335bb16/lb-tcp-51821/0.log: no such file or directory" filename=/var/log/pods/ix-wire-guard-easy_svclb-wire-guard-easy-wg-easy-64xjl_1a6f0098-d7c4-4be6-afd5-4fe4b335bb16/lb-tcp-51821/0.log
2022-12-17T00:21:24.138887695Z level=error ts=2022-12-17T00:21:24.062322397Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/tdarr/1.log: no such file or directory" filename=/var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/tdarr/1.log
2022-12-17T00:21:24.138933648Z level=error ts=2022-12-17T00:21:24.062372001Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-k8s-gateway_svclb-k8s-gateway-6rrww_49f66487-1e62-4d2d-8d88-1f9a653085f1/lb-udp-5353/0.log: no such file or directory" filename=/var/log/pods/ix-k8s-gateway_svclb-k8s-gateway-6rrww_49f66487-1e62-4d2d-8d88-1f9a653085f1/lb-udp-5353/0.log
2022-12-17T00:21:24.138982679Z level=error ts=2022-12-17T00:21:24.062392274Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/tdarr/2.log: no such file or directory" filename=/var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/tdarr/2.log
2022-12-17T00:21:24.139022988Z level=error ts=2022-12-17T00:21:24.062619415Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-lidarr_lidarr-5c4d44956-j876b_3f346cde-5965-4213-a691-31931888c4b4/lidarr/0.log: no such file or directory" filename=/var/log/pods/ix-lidarr_lidarr-5c4d44956-j876b_3f346cde-5965-4213-a691-31931888c4b4/lidarr/0.log
2022-12-17T00:21:24.139060869Z level=error ts=2022-12-17T00:21:24.062755192Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-cookbook_cookbook-ix-chart-85b9c58f97-jvfkz_e676428a-d60a-4f91-b8b1-a9d31469e371/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-cookbook_cookbook-ix-chart-85b9c58f97-jvfkz_e676428a-d60a-4f91-b8b1-a9d31469e371/ix-chart/0.log
2022-12-17T00:21:24.139090412Z level=error ts=2022-12-17T00:21:24.06282835Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27848460-w5znt_d4c27654-6260-41f5-9ccc-8a89b055c148/clamav/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_clam-av-clamav-cronjob-27848460-w5znt_d4c27654-6260-41f5-9ccc-8a89b055c148/clamav/0.log
2022-12-17T00:21:24.139118129Z level=error ts=2022-12-17T00:21:24.062998327Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-tdarr_tdarr-c56cf8bcd-vn2ls_24c123ad-0d0f-42b5-8544-8456c339d7b6/prepare/0.log
2022-12-17T00:21:24.139146447Z level=error ts=2022-12-17T00:21:24.063145589Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/metallb-system_speaker-j4cwx_fc3b0d45-7e4f-493b-9817-2408671909b2/speaker/0.log: no such file or directory" filename=/var/log/pods/metallb-system_speaker-j4cwx_fc3b0d45-7e4f-493b-9817-2408671909b2/speaker/0.log
2022-12-17T00:21:24.139178279Z level=error ts=2022-12-17T00:21:24.063686529Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-wire-guard-easy_svclb-wire-guard-easy-wg-easy-vpn-lbq6x_87141ad4-ae8f-4064-a428-0f15b3040f07/lb-udp-51820/0.log: no such file or directory" filename=/var/log/pods/ix-wire-guard-easy_svclb-wire-guard-easy-wg-easy-vpn-lbq6x_87141ad4-ae8f-4064-a428-0f15b3040f07/lb-udp-51820/0.log
2022-12-17T00:21:24.139218340Z level=error ts=2022-12-17T00:21:24.06374656Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_openebs-zfs-node-4pkqx_5588fd88-4879-4d4f-9cbe-079cc807239a/csi-node-driver-registrar/0.log: no such file or directory" filename=/var/log/pods/kube-system_openebs-zfs-node-4pkqx_5588fd88-4879-4d4f-9cbe-079cc807239a/csi-node-driver-registrar/0.log
2022-12-17T00:21:24.139248825Z level=error ts=2022-12-17T00:21:24.063933996Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_svclb-qbittorrent-torrent-lqd4v_1fc4778b-c580-4449-b066-d1f02cf6a53b/lb-tcp-6881/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_svclb-qbittorrent-torrent-lqd4v_1fc4778b-c580-4449-b066-d1f02cf6a53b/lb-tcp-6881/0.log
2022-12-17T00:21:24.139286810Z level=error ts=2022-12-17T00:21:24.064030545Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-traefik_svclb-traefik-tcp-zfk4t_1928dd1e-ea57-4d58-a4e6-cd48554ff7b8/lb-tcp-443/0.log: no such file or directory" filename=/var/log/pods/ix-traefik_svclb-traefik-tcp-zfk4t_1928dd1e-ea57-4d58-a4e6-cd48554ff7b8/lb-tcp-443/0.log
2022-12-17T00:21:24.139325350Z level=error ts=2022-12-17T00:21:24.064306277Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-qbittorrent_svclb-qbittorrent-s98rx_86f57ba0-1663-4902-8252-41d377571476/lb-tcp-10095/0.log: no such file or directory" filename=/var/log/pods/ix-qbittorrent_svclb-qbittorrent-s98rx_86f57ba0-1663-4902-8252-41d377571476/lb-tcp-10095/0.log
2022-12-17T00:21:24.139361957Z level=error ts=2022-12-17T00:21:24.064351987Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-nweb-db_nweb-db-ix-chart-5dbf7ccd4b-mx8kx_65c3a614-b6ad-4fa3-9d78-c69af7f1b6bb/ix-chart/0.log: no such file or directory" filename=/var/log/pods/ix-nweb-db_nweb-db-ix-chart-5dbf7ccd4b-mx8kx_65c3a614-b6ad-4fa3-9d78-c69af7f1b6bb/ix-chart/0.log
2022-12-17T00:21:24.139410306Z level=error ts=2022-12-17T00:21:24.064445802Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-loki-new_loki-new-promtail-54465f6787-rml5p_e2338fa3-5b64-4054-9167-ed271dda63f9/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-loki-new_loki-new-promtail-54465f6787-rml5p_e2338fa3-5b64-4054-9167-ed271dda63f9/prepare/0.log
2022-12-17T00:21:24.139455572Z level=error ts=2022-12-17T00:21:24.064988747Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-hades_hades-plex-7dd7d446d6-58jc6_f395c096-1c0d-4364-9f57-cc1b696a3e36/hades-plex/0.log: no such file or directory" filename=/var/log/pods/ix-hades_hades-plex-7dd7d446d6-58jc6_f395c096-1c0d-4364-9f57-cc1b696a3e36/hades-plex/0.log
2022-12-17T00:21:24.139501262Z level=error ts=2022-12-17T00:21:24.065068435Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-lidarr_lidarr-5c4d44956-j876b_3f346cde-5965-4213-a691-31931888c4b4/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-lidarr_lidarr-5c4d44956-j876b_3f346cde-5965-4213-a691-31931888c4b4/prepare/0.log
2022-12-17T00:21:24.139545877Z level=error ts=2022-12-17T00:21:24.065184999Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-jackett_svclb-jackett-8z8bw_7a322ec0-1b7f-4d3c-a919-24be1dd4fd44/lb-tcp-9117/0.log: no such file or directory" filename=/var/log/pods/ix-jackett_svclb-jackett-8z8bw_7a322ec0-1b7f-4d3c-a919-24be1dd4fd44/lb-tcp-9117/0.log
2022-12-17T00:21:24.139642562Z level=error ts=2022-12-17T00:21:24.065242404Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-audio-bookshelf_audio-bookshelf-audiobookshelf-567ff998c7-jdgz8_0c18628b-082c-469c-80b7-870aac061832/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-audio-bookshelf_audio-bookshelf-audiobookshelf-567ff998c7-jdgz8_0c18628b-082c-469c-80b7-870aac061832/prepare/0.log
2022-12-17T00:21:24.139692537Z level=error ts=2022-12-17T00:21:24.065281546Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-clam-av_svclb-clam-av-clamav-vhdvc_a0112326-d3e2-4097-8c6d-5c7018ecfe95/lb-tcp-3310/0.log: no such file or directory" filename=/var/log/pods/ix-clam-av_svclb-clam-av-clamav-vhdvc_a0112326-d3e2-4097-8c6d-5c7018ecfe95/lb-tcp-3310/0.log
2022-12-17T00:21:24.139737690Z level=error ts=2022-12-17T00:21:24.065402285Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-sonarr_svclb-sonarr-tzjwr_cab92b72-7706-4f58-9108-96b86925b1c6/lb-tcp-8989/0.log: no such file or directory" filename=/var/log/pods/ix-sonarr_svclb-sonarr-tzjwr_cab92b72-7706-4f58-9108-96b86925b1c6/lb-tcp-8989/0.log
2022-12-17T00:21:24.139769686Z level=error ts=2022-12-17T00:21:24.06548019Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-prowlarr_prowlarr-5d987446d8-zcvjk_e851584f-14a5-4fc1-aedc-a52466d39a21/prepare/0.log: no such file or directory" filename=/var/log/pods/ix-prowlarr_prowlarr-5d987446d8-zcvjk_e851584f-14a5-4fc1-aedc-a52466d39a21/prepare/0.log
2022-12-17T00:21:24.139805238Z level=error ts=2022-12-17T00:21:24.065616919Z caller=filetarget.go:311 msg="failed to tail file, stat failed" error="stat /var/log/pods/ix-audio-bookshelf_svclb-audio-bookshelf-audiobookshelf-654cv_4c2e2f13-3b33-43d4-9348-0edfe8a9759f/lb-tcp-10223/0.log: no such file or directory" filename=/var/log/pods/ix-audio-bookshelf_svclb-audio-bookshelf-audiobookshelf-654cv_4c2e2f13-3b33-43d4-9348-0edfe8a9759f/lb-tcp-10223/0.log


loki-0 logs:
2022-12-17T00:06:14.446062912Z level=info ts=2022-12-17T00:06:14.349266187Z caller=main.go:103 msg="Starting Loki" version="(version=2.7.1, branch=HEAD, revision=e0af1cc8a)"
2022-12-17T00:06:14.446125426Z level=info ts=2022-12-17T00:06:14.349465939Z caller=modules.go:862 msg="Ruler storage is not configured; ruler will not be started."
2022-12-17T00:06:14.446132353Z level=info ts=2022-12-17T00:06:14.350775222Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
2022-12-17T00:06:14.446141374Z level=warn ts=2022-12-17T00:06:14.352247981Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
2022-12-17T00:06:14.446146814Z level=warn ts=2022-12-17T00:06:14.352519877Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
2022-12-17T00:06:14.446154637Z level=info ts=2022-12-17T00:06:14.353413058Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:06:14.446159862Z level=info ts=2022-12-17T00:06:14.353562715Z caller=table_manager.go:262 msg="query readiness setup completed" duration=1.253µs distinct_users_len=0
2022-12-17T00:06:14.446165406Z level=info ts=2022-12-17T00:06:14.353660927Z caller=shipper.go:127 msg="starting index shipper in RW mode"
2022-12-17T00:06:14.446172900Z level=info ts=2022-12-17T00:06:14.353880405Z caller=shipper_index_client.go:78 msg="starting boltdb shipper in RW mode"
2022-12-17T00:06:14.446178175Z level=info ts=2022-12-17T00:06:14.353952605Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:06:14.446183293Z level=info ts=2022-12-17T00:06:14.360472084Z caller=modules.go:881 msg="RulerStorage is nil.  Not starting the ruler."
2022-12-17T00:06:14.446190865Z level=info ts=2022-12-17T00:06:14.361120157Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
2022-12-17T00:06:14.446196180Z level=info ts=2022-12-17T00:06:14.365299352Z caller=module_service.go:82 msg=initialising module=cache-generation-loader
2022-12-17T00:06:14.446203942Z level=info ts=2022-12-17T00:06:14.365316925Z caller=module_service.go:82 msg=initialising module=server
2022-12-17T00:06:14.446209113Z level=info ts=2022-12-17T00:06:14.365345692Z caller=module_service.go:82 msg=initialising module=memberlist-kv
2022-12-17T00:06:14.446214114Z level=info ts=2022-12-17T00:06:14.365350107Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
2022-12-17T00:06:14.446222758Z level=info ts=2022-12-17T00:06:14.365420027Z caller=module_service.go:82 msg=initialising module=ring
2022-12-17T00:06:14.446227990Z level=info ts=2022-12-17T00:06:14.365436931Z caller=module_service.go:82 msg=initialising module=store
2022-12-17T00:06:14.446233021Z level=info ts=2022-12-17T00:06:14.365746682Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
2022-12-17T00:06:14.446240954Z level=info ts=2022-12-17T00:06:14.365832438Z caller=client.go:255 msg="value is nil" key=collectors/ring index=1
2022-12-17T00:06:14.446246273Z level=info ts=2022-12-17T00:06:14.365846862Z caller=module_service.go:82 msg=initialising module=usage-report
2022-12-17T00:06:14.446251370Z level=info ts=2022-12-17T00:06:14.365850462Z caller=module_service.go:82 msg=initialising module=ingester-querier
2022-12-17T00:06:14.446259427Z level=info ts=2022-12-17T00:06:14.365940442Z caller=module_service.go:82 msg=initialising module=distributor
2022-12-17T00:06:14.446264604Z level=info ts=2022-12-17T00:06:14.365942651Z caller=module_service.go:82 msg=initialising module=compactor
2022-12-17T00:06:14.446269612Z level=info ts=2022-12-17T00:06:14.365957794Z caller=module_service.go:82 msg=initialising module=query-scheduler
2022-12-17T00:06:14.446277165Z level=info ts=2022-12-17T00:06:14.365983077Z caller=module_service.go:82 msg=initialising module=ingester
2022-12-17T00:06:14.446282236Z level=info ts=2022-12-17T00:06:14.366084073Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
2022-12-17T00:06:14.446287347Z level=info ts=2022-12-17T00:06:14.366094667Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
2022-12-17T00:06:14.446295032Z level=info ts=2022-12-17T00:06:14.366094629Z caller=ingester.go:417 msg="recovering from checkpoint"
2022-12-17T00:06:14.446300262Z level=info ts=2022-12-17T00:06:14.36612779Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
2022-12-17T00:06:14.446307806Z level=info ts=2022-12-17T00:06:14.366192011Z caller=recovery.go:39 msg="no checkpoint found, treating as no-op"
2022-12-17T00:06:14.446313025Z level=info ts=2022-12-17T00:06:14.366239128Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=distributor
2022-12-17T00:06:14.446319168Z level=info ts=2022-12-17T00:06:14.366122547Z caller=client.go:255 msg="value is nil" key=collectors/scheduler index=1
2022-12-17T00:06:14.446329123Z level=info ts=2022-12-17T00:06:14.366132607Z caller=client.go:255 msg="value is nil" key=collectors/compactor index=1
2022-12-17T00:06:14.446334828Z level=info ts=2022-12-17T00:06:14.366279007Z caller=ingester.go:433 msg="recovered WAL checkpoint recovery finished" elapsed=253.486µs errors=false
2022-12-17T00:06:14.446342642Z level=info ts=2022-12-17T00:06:14.366300148Z caller=ingester.go:439 msg="recovering from WAL"
2022-12-17T00:06:14.446347868Z level=info ts=2022-12-17T00:06:14.366358353Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=loki-new-0 ring=compactor
2022-12-17T00:06:14.446353026Z level=info ts=2022-12-17T00:06:14.366359367Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=loki-new-0 ring=scheduler
2022-12-17T00:06:14.446361039Z level=info ts=2022-12-17T00:06:14.366390108Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
2022-12-17T00:06:14.446366299Z level=info ts=2022-12-17T00:06:14.366390542Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
2022-12-17T00:06:14.446374109Z level=info ts=2022-12-17T00:06:14.366487144Z caller=ingester.go:455 msg="WAL segment recovery finished" elapsed=461.751µs errors=false
2022-12-17T00:06:14.446380253Z level=info ts=2022-12-17T00:06:14.366499306Z caller=compactor.go:328 msg="waiting until compactor is JOINING in the ring"
2022-12-17T00:06:14.446388420Z level=info ts=2022-12-17T00:06:14.366507495Z caller=ingester.go:403 msg="closing recoverer"
2022-12-17T00:06:14.446400142Z level=info ts=2022-12-17T00:06:14.366524022Z caller=ingester.go:411 msg="WAL recovery finished" time=498.085µs
2022-12-17T00:06:14.446406001Z level=info ts=2022-12-17T00:06:14.366585619Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=distributor
2022-12-17T00:06:14.446411145Z level=info ts=2022-12-17T00:06:14.366600663Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
2022-12-17T00:06:14.446419378Z level=info ts=2022-12-17T00:06:14.366634549Z caller=wal.go:156 msg=started component=wal
2022-12-17T00:06:14.446424568Z level=info ts=2022-12-17T00:06:14.366714492Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
2022-12-17T00:06:14.446432361Z level=info ts=2022-12-17T00:06:14.3667973Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
2022-12-17T00:06:14.446437645Z level=info ts=2022-12-17T00:06:14.366794716Z caller=scheduler.go:617 msg="waiting until scheduler is JOINING in the ring"
2022-12-17T00:06:14.446443053Z level=info ts=2022-12-17T00:06:14.366831318Z caller=scheduler.go:621 msg="scheduler is JOINING in the ring"
2022-12-17T00:06:14.446450665Z level=info ts=2022-12-17T00:06:14.366888727Z caller=scheduler.go:631 msg="waiting until scheduler is ACTIVE in the ring"
2022-12-17T00:06:14.546391745Z level=info ts=2022-12-17T00:06:14.484152096Z caller=compactor.go:332 msg="compactor is JOINING in the ring"
2022-12-17T00:06:14.546436852Z level=info ts=2022-12-17T00:06:14.48430581Z caller=compactor.go:342 msg="waiting until compactor is ACTIVE in the ring"
2022-12-17T00:06:14.646558851Z level=info ts=2022-12-17T00:06:14.553117926Z caller=scheduler.go:635 msg="scheduler is ACTIVE in the ring"
2022-12-17T00:06:14.646592952Z level=info ts=2022-12-17T00:06:14.553176019Z caller=module_service.go:82 msg=initialising module=query-frontend
2022-12-17T00:06:14.646602799Z level=info ts=2022-12-17T00:06:14.553212136Z caller=module_service.go:82 msg=initialising module=querier
2022-12-17T00:06:14.646620387Z level=info ts=2022-12-17T00:06:14.600299601Z caller=compactor.go:346 msg="compactor is ACTIVE in the ring"
2022-12-17T00:06:14.646628966Z level=info ts=2022-12-17T00:06:14.600357059Z caller=loki.go:402 msg="Loki started"
2022-12-17T00:06:17.646442121Z level=info ts=2022-12-17T00:06:17.553551693Z caller=scheduler.go:682 msg="this scheduler is in the ReplicationSet, will now accept requests."
2022-12-17T00:06:17.646486039Z level=info ts=2022-12-17T00:06:17.553672789Z caller=worker.go:209 msg="adding connection" addr=172.16.5.179:9095
2022-12-17T00:06:19.645968234Z level=info ts=2022-12-17T00:06:19.600830733Z caller=compactor.go:407 msg="this instance has been chosen to run the compactor, starting compactor"
2022-12-17T00:06:19.646028463Z level=info ts=2022-12-17T00:06:19.600904013Z caller=compactor.go:436 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"
2022-12-17T00:06:24.646011016Z level=info ts=2022-12-17T00:06:24.554129823Z caller=frontend_scheduler_worker.go:101 msg="adding connection to scheduler" addr=172.16.5.179:9095
2022-12-17T00:07:14.446875950Z level=info ts=2022-12-17T00:07:14.353597897Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:07:14.446915164Z level=info ts=2022-12-17T00:07:14.354760785Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:08:14.446308375Z level=info ts=2022-12-17T00:08:14.353558462Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:08:14.446356826Z level=info ts=2022-12-17T00:08:14.354997664Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:09:14.446206189Z level=info ts=2022-12-17T00:09:14.354045611Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:09:14.446248128Z level=info ts=2022-12-17T00:09:14.354143482Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:10:14.446120646Z level=info ts=2022-12-17T00:10:14.353495398Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:10:14.446155937Z level=info ts=2022-12-17T00:10:14.354702666Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:11:14.446002689Z level=info ts=2022-12-17T00:11:14.353639155Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:11:14.446051218Z level=info ts=2022-12-17T00:11:14.354214141Z caller=table_manager.go:223 msg="syncing tables"
2022-12-17T00:11:14.446057749Z level=info ts=2022-12-17T00:11:14.354272389Z caller=table_manager.go:262 msg="query readiness setup completed" duration=2.98µs distinct_users_len=0
2022-12-17T00:11:14.446063423Z level=info ts=2022-12-17T00:11:14.354849379Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:11:14.446072552Z level=info ts=2022-12-17T00:11:14.367519003Z caller=checkpoint.go:614 msg="starting checkpoint"
2022-12-17T00:11:14.446077758Z level=info ts=2022-12-17T00:11:14.368329892Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.000000
2022-12-17T00:11:14.446082772Z level=info ts=2022-12-17T00:11:14.403533126Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=/data/loki/wal/checkpoint.000000.tmp new=/data/loki/wal/checkpoint.000000
2022-12-17T00:12:14.446727636Z level=info ts=2022-12-17T00:12:14.35355607Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:12:14.446766504Z level=info ts=2022-12-17T00:12:14.354769642Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:13:14.446088970Z level=info ts=2022-12-17T00:13:14.354330244Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:13:14.446130663Z level=info ts=2022-12-17T00:13:14.354359532Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:14:14.446858668Z level=info ts=2022-12-17T00:14:14.353499124Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:14:14.446995677Z level=info ts=2022-12-17T00:14:14.354685641Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:15:14.446726761Z level=info ts=2022-12-17T00:15:14.354320283Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:15:14.446906978Z level=info ts=2022-12-17T00:15:14.354356011Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:16:14.446328574Z level=info ts=2022-12-17T00:16:14.354499566Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:16:14.446368053Z level=info ts=2022-12-17T00:16:14.35455482Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:16:14.446375149Z level=info ts=2022-12-17T00:16:14.354552423Z caller=table_manager.go:223 msg="syncing tables"
2022-12-17T00:16:14.446392435Z level=info ts=2022-12-17T00:16:14.354592828Z caller=table_manager.go:262 msg="query readiness setup completed" duration=2.388µs distinct_users_len=0
2022-12-17T00:16:14.446398743Z level=info ts=2022-12-17T00:16:14.366898519Z caller=checkpoint.go:614 msg="starting checkpoint"
2022-12-17T00:16:14.446404210Z level=info ts=2022-12-17T00:16:14.367178883Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.000001
2022-12-17T00:16:14.446412630Z level=info ts=2022-12-17T00:16:14.387877308Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=/data/loki/wal/checkpoint.000001.tmp new=/data/loki/wal/checkpoint.000001
2022-12-17T00:16:19.646201340Z level=info ts=2022-12-17T00:16:19.601976724Z caller=compactor.go:441 msg="compactor startup delay completed"
2022-12-17T00:16:19.646255238Z level=info ts=2022-12-17T00:16:19.602044267Z caller=compactor.go:493 msg="compactor started"
2022-12-17T00:16:19.646265926Z ts=2022-12-17T00:16:19.604198668Z caller=spanlogger.go:80 level=info msg="building index list cache"
2022-12-17T00:16:19.646274269Z ts=2022-12-17T00:16:19.606647958Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=89.667µs
2022-12-17T00:17:14.446238242Z level=info ts=2022-12-17T00:17:14.354369028Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:17:14.446300709Z level=info ts=2022-12-17T00:17:14.354443199Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:18:14.446464004Z level=info ts=2022-12-17T00:18:14.354257181Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:18:14.446519221Z level=info ts=2022-12-17T00:18:14.354305534Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:19:14.446226979Z level=info ts=2022-12-17T00:19:14.354078546Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:19:14.446339482Z level=info ts=2022-12-17T00:19:14.354124105Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:20:14.446822688Z level=info ts=2022-12-17T00:20:14.35352637Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:20:14.446857923Z level=info ts=2022-12-17T00:20:14.354725714Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:21:14.446259292Z level=info ts=2022-12-17T00:21:14.353806548Z caller=table_manager.go:223 msg="syncing tables"
2022-12-17T00:21:14.446304145Z level=info ts=2022-12-17T00:21:14.353843139Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:21:14.446312677Z level=info ts=2022-12-17T00:21:14.35387868Z caller=table_manager.go:262 msg="query readiness setup completed" duration=3.231µs distinct_users_len=0
2022-12-17T00:21:14.446332035Z level=info ts=2022-12-17T00:21:14.355142763Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:21:14.446338690Z level=info ts=2022-12-17T00:21:14.366743903Z caller=checkpoint.go:614 msg="starting checkpoint"
2022-12-17T00:21:14.446344806Z level=info ts=2022-12-17T00:21:14.367210184Z caller=checkpoint.go:339 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.000002
2022-12-17T00:21:14.446355546Z level=info ts=2022-12-17T00:21:14.423814628Z caller=checkpoint.go:501 msg="atomic checkpoint finished" old=/data/loki/wal/checkpoint.000002.tmp new=/data/loki/wal/checkpoint.000002
2022-12-17T00:22:14.446303460Z level=info ts=2022-12-17T00:22:14.354303692Z caller=table_manager.go:166 msg="handing over indexes to shipper"
2022-12-17T00:22:14.446359787Z level=info ts=2022-12-17T00:22:14.354351748Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:23:14.446446005Z level=info ts=2022-12-17T00:23:14.353522529Z caller=table_manager.go:134 msg="uploading tables"
2022-12-17T00:23:14.446504556Z level=info ts=2022-12-17T00:23:14.354642371Z caller=table_manager.go:166 msg="handing over indexes to shipper"

Application Configuration

image image image image

Describe the bug

  1. Logs are not being received by Grafana when loki is added as a data source. The last working version for me was 2.7.0_7.0.0. All versions past that exhibit the same behavior, even with a fresh install. It appears that there is some kind of internal error in loki-promtail that is preventing logs from being scraped which is resulting in nothing being received by Grafana.

To Reproduce

Install the loki app from Truecharts using the default settings.

Expected Behavior

Grafana should be receiving logs from loki app when added as a data source

Screenshots

When pointing Grafana to loki as a data source I get this error message when using any version or the Truecharts loki app higher than version 2.7.0_7.0.0 even when using a fresh install.

image

Additional Context

This is the same issue as #5384, but re-submitting after confirming to still be an issue when using a new install as requested.

I've read and agree with the following

  • [X] I've checked all open and closed issues and my issue is not there.

NickScarpitti avatar Dec 17 '22 00:12 NickScarpitti

yep same here, I opened a ticket inside TC discord server, it's ticket-361933593288769536 and I shared some information there. still no solution: at the moment grafana+prometheus+loki is unusable

mariopaolo avatar Dec 20 '22 23:12 mariopaolo

Issue is reported for 22.02.3, which is not in the supported versions list…

PrivatePuffin avatar Dec 21 '22 09:12 PrivatePuffin

@Ornias1993 you can reproduce it with latest version for any app mentioned on Bluefin. also check my ticket in Discord if you need more info.

mariopaolo avatar Dec 21 '22 11:12 mariopaolo

Thanks for confirming, i'll reopen. However, please post ALL required info about bug here.

Support tickets are not read for development purposes.

PrivatePuffin avatar Dec 21 '22 13:12 PrivatePuffin

I tested the steps to reproduce the issue on TrueNAS SCALE Bluefin 22.12

  • grafana 9.3.2_6.0.17
  • loki 2.7.1_9.0.5

both deployed from scratch on a working SCALE install, with all defaults (in Bluefin it's just one click since you don't need to skip through the various sections). I then went to add loki as a source in grafana: using standard naming for TC apps, loki is reachable at http://loki.ix-loki.svc.cluster.local:3100.

I can confirm I get the exact same results as @NickScarpitti image

Applications Events loki

2022-12-21 18:38:09
Error: Error response from daemon: invalid volume specification: '/var/log/pods:/var/log/pods:ro': /var/db/system/syslog-535a85bf35cc472d88ee664a1c80b367/log/pods path (real path of /var/log/pods) not allowed to be mounted
2022-12-21 18:37:58
Successfully pulled image "tccr.io/truecharts/loki:2.7.1@sha256:a94fe45062e6dfefeb9b9363e266744a6b43449f47521f8676d755462be2d872" in 5.611638144s
2022-12-21 18:37:58
Created container loki
2022-12-21 18:37:58
Started container loki
2022-12-21 18:37:54
Successfully pulled image "ghcr.io/truecharts/promtail:2.7.1@sha256:94be035cd342c5d4a088394fd68e3bb3f1071eeb1ab20aeb1886b7f14d842cce" in 3.568374164s
2022-12-21 18:37:54
Error: Error response from daemon: invalid volume specification: '/var/log/pods:/var/log/pods:ro': /var/log/pods path not allowed to be mounted
2022-12-21 18:37:54
Container image "ghcr.io/truecharts/promtail:2.7.1@sha256:94be035cd342c5d4a088394fd68e3bb3f1071eeb1ab20aeb1886b7f14d842cce" already present on machine
2022-12-21 18:37:52
Add eth0 [172.16.0.251/16] from ix-net
2022-12-21 18:37:52
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:37:52
Created container prepare
2022-12-21 18:37:52
Started container prepare
2022-12-21 18:37:52
Pulling image "tccr.io/truecharts/loki:2.7.1@sha256:a94fe45062e6dfefeb9b9363e266744a6b43449f47521f8676d755462be2d872"
2022-12-21 18:37:50
Add eth0 [172.16.0.250/16] from ix-net
2022-12-21 18:37:50
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:37:50
Created container prepare
2022-12-21 18:37:50
Started container prepare
2022-12-21 18:37:50
Pulling image "ghcr.io/truecharts/promtail:2.7.1@sha256:94be035cd342c5d4a088394fd68e3bb3f1071eeb1ab20aeb1886b7f14d842cce"
2022-12-21 18:37:50
Updated LoadBalancer with new IPs: [] -> [192.168.1.139]
2022-12-21 18:37:50
Successfully provisioned volume pvc-939030aa-77e0-403e-96bc-a62df962a0ba
2022-12-21 18:37:50
Successfully assigned ix-loki/loki-0 to ix-truenas
2022-12-21 18:37:49
Job completed
2022-12-21 18:37:49
Ensuring load balancer
2022-12-21 18:37:49
Scaled up replica set loki-promtail-68f6fc459f to 1
2022-12-21 18:37:49
Created pod: loki-promtail-68f6fc459f-s85q2
2022-12-21 18:37:49
Applied LoadBalancer DaemonSet kube-system/svclb-loki-081cb87c
2022-12-21 18:37:49
Successfully assigned ix-loki/loki-promtail-68f6fc459f-s85q2 to ix-truenas
2022-12-21 18:37:49
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
2022-12-21 18:37:49
External provisioner is provisioning volume for claim "ix-loki/data-loki-0"
2022-12-21 18:37:49
create Claim data-loki-0 Pod loki-0 in StatefulSet loki success
2022-12-21 18:37:49
create Pod loki-0 in StatefulSet loki successful
2022-12-21 18:37:49
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-12-21 18:37:43
Add eth0 [172.16.0.248/16] from ix-net
2022-12-21 18:37:43
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:37:43
Created container loki-promtail-manifests
2022-12-21 18:37:43
Started container loki-promtail-manifests
2022-12-21 18:37:42
Job completed
2022-12-21 18:37:42
Created pod: loki-promtail-manifests-mcdzx
2022-12-21 18:37:42
Successfully assigned ix-loki/loki-promtail-manifests-mcdzx to ix-truenas
2022-12-21 18:37:36
Add eth0 [172.16.0.247/16] from ix-net
2022-12-21 18:37:36
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:37:36
Created container loki-manifests
2022-12-21 18:37:36
Started container loki-manifests
2022-12-21 18:37:35
Created pod: loki-manifests-nqlgb
2022-12-21 18:37:35
Successfully assigned ix-loki/loki-manifests-nqlgb to ix-truenas

grafana

2022-12-21 18:39:01
Container image "tccr.io/truecharts/grafana:9.3.2@sha256:ed6cf9e00a2bfcdcc8e54d83e7c0547d9a00fe9cbdeade15c2b32e36484d9b80" already present on machine
2022-12-21 18:39:01
Created container grafana
2022-12-21 18:39:01
Started container grafana
2022-12-21 18:39:00
Add eth0 [172.16.0.254/16] from ix-net
2022-12-21 18:39:00
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:39:00
Created container prepare
2022-12-21 18:39:00
Started container prepare
2022-12-21 18:38:59
Successfully assigned ix-grafana/grafana-6d7d949b65-z2k7g to ix-truenas
2022-12-21 18:38:57
Updated LoadBalancer with new IPs: [] -> [192.168.1.139]
2022-12-21 18:38:57
Successfully provisioned volume pvc-1beb4aae-2da5-4226-bb64-5feacef38f4b
2022-12-21 18:38:57
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
2022-12-21 18:38:56
Job completed
2022-12-21 18:38:56
External provisioner is provisioning volume for claim "ix-grafana/grafana-config"
2022-12-21 18:38:56
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-12-21 18:38:56
Ensuring load balancer
2022-12-21 18:38:56
Applied LoadBalancer DaemonSet kube-system/svclb-grafana-58dfee8f
2022-12-21 18:38:56
Scaled up replica set grafana-6d7d949b65 to 1
2022-12-21 18:38:56
Created pod: grafana-6d7d949b65-z2k7g
2022-12-21 18:38:56
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
2022-12-21 18:38:49
Created pod: grafana-manifests-6s475
2022-12-21 18:38:49
Successfully assigned ix-grafana/grafana-manifests-6s475 to ix-truenas
2022-12-21 18:38:49
Add eth0 [172.16.0.252/16] from ix-net
2022-12-21 18:38:49
Container image "tccr.io/truecharts/ubuntu:jammy-20221101@sha256:4b9475e08c5180d4e7417dc6a18a26dcce7691e4311e5353dbb952645c5ff43f" already present on machine
2022-12-21 18:38:49
Created container grafana-manifests
2022-12-21 18:38:49
Started container grafana-manifests

P.S.: this error

Error: Error response from daemon: invalid volume specification: '/var/log/pods:/var/log/pods:ro': /var/db/system/syslog-535a85bf35cc472d88ee664a1c80b367/log/pods path (real path of /var/log/pods) not allowed to be mounted

inside loki applications event is new and reproducible, but for now outside the scope since the issue described here has existed before this error materialized in newer versions. also this error doesn't prevent loki from starting.

mariopaolo avatar Dec 21 '22 17:12 mariopaolo

As the "path not allowed to be mounted" is the only actually traceable error at the moment on the current version, i'm going to close this. As this is caused by hostPath validation (most likely) and not maintained by us.

As soon as that upstream change is fixed, we can look into this again.

PrivatePuffin avatar Dec 21 '22 18:12 PrivatePuffin

While it can be worked on by disabling hostPath validation for now, it cannot actually be run with that disabled due to our support scope (aka: no hostPath validation == no support and no bugfixes either).

Hence this specific issue is blocked until a fix for this is released: https://ixsystems.atlassian.net/browse/NAS-119521

PrivatePuffin avatar Dec 21 '22 19:12 PrivatePuffin

I was able to get Loki up and ingesting logs with the promtail instance by adding a mount to promtail for where /var/log/pods eventually points: /mnt/[POOL]/ix-applications/docker/containers

I had to edit the deployment for promtail, but once that was done the issue was resolved and grafana was able to complete a local test

seanb4t avatar Dec 23 '22 14:12 seanb4t

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

truecharts-admin avatar Jun 21 '23 20:06 truecharts-admin

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

truecharts-admin avatar Feb 20 '24 20:02 truecharts-admin

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

truecharts-admin avatar Mar 06 '24 20:03 truecharts-admin