helm-charts icon indicating copy to clipboard operation
helm-charts copied to clipboard

Published 20 hours ago • verified publisher icon grafana

[loki-stack] loki pod crashing with panic opening boltdb file

Open saigha01 opened this issue 2 years ago • 0 comments

loki sts pod crashing with panic opening boltdb file

Helm chart:

`$ helm show chart grafana/loki-stack apiVersion: v1 appVersion: v2.4.2 dependencies:

  • condition: loki.enabled name: loki repository: https://grafana.github.io/helm-charts version: ^2.14.1
  • condition: promtail.enabled name: promtail repository: https://grafana.github.io/helm-charts version: ^6.2.3
  • condition: fluent-bit.enabled name: fluent-bit repository: https://grafana.github.io/helm-charts version: ^2.3.0
  • condition: grafana.enabled name: grafana repository: https://grafana.github.io/helm-charts version: ~6.24.1
  • condition: prometheus.enabled name: prometheus repository: https://prometheus-community.github.io/helm-charts version: ~15.5.3
  • condition: filebeat.enabled name: filebeat repository: https://helm.elastic.co version: ~7.17.1
  • condition: logstash.enabled name: logstash repository: https://helm.elastic.co version: ~7.17.1 description: 'Loki: like Prometheus, but for logs.' home: https://grafana.com/loki icon: https://raw.githubusercontent.com/grafana/loki/master/docs/sources/logo.png kubeVersion: ^1.10.0-0 maintainers:
  • email: [email protected] name: Loki Maintainers name: loki-stack sources:
  • https://github.com/grafana/loki version: 2.7.0 `

status of loki-stack deployment: `$ kubectl get all -n loki-stack NAME READY STATUS RESTARTS AGE pod/loki-stack-0 0/1 CrashLoopBackOff 7 (43s ago) 11m pod/loki-stack-grafana-7d8859f94b-vktg5 2/2 Running 0 46h pod/loki-stack-promtail-76x4s 1/1 Running 0 46h pod/loki-stack-promtail-86gtw 1/1 Running 0 46h pod/loki-stack-promtail-8lldc 1/1 Running 0 46h pod/loki-stack-promtail-9jx2x 1/1 Running 0 46h pod/loki-stack-promtail-kcjdh 1/1 Running 0 46h pod/loki-stack-promtail-t5fmk 1/1 Running 0 46h pod/loki-stack-promtail-tm5v8 1/1 Running 0 46h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/loki-stack ClusterIP 10.233.42.121 3100/TCP 46h service/loki-stack-grafana ClusterIP 10.233.45.91 80/TCP 46h service/loki-stack-headless ClusterIP None 3100/TCP 46h service/loki-stack-memberlist ClusterIP None 7946/TCP 46h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/loki-stack-promtail 7 7 7 7 7 46h

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/loki-stack-grafana 1/1 1 1 46h

NAME DESIRED CURRENT READY AGE replicaset.apps/loki-stack-grafana-7d8859f94b 1 1 1 46h

NAME READY AGE statefulset.apps/loki-stack 0/1 46h`

PV and PVC for loki-stack:

`$ kubectl get pvc -n loki-stack NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE loki-stack-grafana Bound pvc-858ee02c-3c3c-4dc3-b91c-9d4af013f425 20Gi RWO infra-nfs-client 46h storage-loki-stack-0 Bound pvc-aabf91ad-a31c-4eca-a3bd-a9af116ba038 50Gi RWO infra-nfs-client 20h

$ kubectl get pv -n loki-stack NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-858ee02c-3c3c-4dc3-b91c-9d4af013f425 20Gi RWO Delete Bound loki-stack/loki-stack-grafana infra-nfs-client 46h pvc-aabf91ad-a31c-4eca-a3bd-a9af116ba038 50Gi RWO Delete Bound loki-stack/storage-loki-stack-0 infra-nfs-client 20h

$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE infra-nfs-client infra-nfs-client Delete Immediate false 28d`

We are using nfs storage class and expecting loki to store logs sent from prometheus. When the setup is done, for first few hours it was running perfectly but after than we started noticing 502 gateway error. When looked in loki installation, we noticed the loki pod is been in crashloopBackoff. When looked into logs, we found the following log.

$ kubectl logs loki-stack-0 -n loki-stack level=info ts=2022-08-26T08:32:31.485320333Z caller=main.go:103 msg="Starting Loki" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)" level=info ts=2022-08-26T08:32:31.485696395Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses" level=info ts=2022-08-26T08:32:31.485939931Z caller=modules.go:736 msg="RulerStorage is not configured in single binary mode and will not be started." level=warn ts=2022-08-26T08:32:31.487734144Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksfifocache" level=info ts=2022-08-26T08:32:31.489919848Z caller=table_manager.go:381 msg="loading local table index_19229" level=info ts=2022-08-26T08:32:31.491426791Z caller=table_manager.go:134 msg="uploading tables" level=error ts=2022-08-26T08:32:31.493131986Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661428658, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: invalid freelist page: 0, page type is unknown<00>" level=error ts=2022-08-26T08:32:31.495272565Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661428800, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: invalid freelist page: 0, page type is unknown<00>" level=info ts=2022-08-26T08:32:31.495617903Z caller=memberlist_client.go:534 msg="joined memberlist cluster" reached_nodes=1 level=error ts=2022-08-26T08:32:31.497261697Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661429700, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.499137314Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661430600, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.500859706Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661431500, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.502923762Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661432400, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.504985627Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661433300, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: invalid freelist page: 0, page type is meta" level=error ts=2022-08-26T08:32:31.506622563Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661434200, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.508476964Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661435100, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.510098854Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661436000, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.511669983Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661436900, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.51334169Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661437800, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: invalid freelist page: 0, page type is meta" level=error ts=2022-08-26T08:32:31.515016111Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661438700, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.51650413Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661439600, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.518040504Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661440500, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.519497843Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661441400, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: invalid freelist page: 1, page type is meta" level=error ts=2022-08-26T08:32:31.520988273Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661442300, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.522397056Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661443200, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.523984133Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661444100, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.526767021Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661445000, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.528346868Z caller=index_set.go:133 table-name=index_19229 msg="failed to open existing index file /data/loki/boltdb-shipper-cache/index_19229/loki-stack-0-1661428388368789724-1661445900, removing the file and continuing without it to let the sync operation catch up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" ts=2022-08-26T08:32:31.529145975Z caller=spanlogger.go:80 level=info msg="building index list cache" ts=2022-08-26T08:32:31.532354518Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=3.191678ms level=info ts=2022-08-26T08:32:31.535228904Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661428658.gz msg="downloaded file" total_time=2.682219ms level=info ts=2022-08-26T08:32:31.536772762Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661428800.gz msg="downloaded file" total_time=4.229555ms level=info ts=2022-08-26T08:32:31.537698039Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661429700.gz msg="downloaded file" total_time=5.132249ms level=info ts=2022-08-26T08:32:31.538411256Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661430600.gz msg="downloaded file" total_time=5.804155ms level=info ts=2022-08-26T08:32:31.539501988Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661431500.gz msg="downloaded file" total_time=2.057956ms level=info ts=2022-08-26T08:32:31.54018669Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661432400.gz msg="downloaded file" total_time=2.05937ms level=info ts=2022-08-26T08:32:31.541220989Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661433300.gz msg="downloaded file" total_time=2.12707ms level=info ts=2022-08-26T08:32:31.541767553Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661436900.gz msg="downloaded file" total_time=2.369697ms level=info ts=2022-08-26T08:32:31.542737802Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661436000.gz msg="downloaded file" total_time=3.397967ms level=info ts=2022-08-26T08:32:31.543475427Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661434200.gz msg="downloaded file" total_time=4.299413ms level=info ts=2022-08-26T08:32:31.544310975Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661435100.gz msg="downloaded file" total_time=5.009461ms level=info ts=2022-08-26T08:32:31.54491828Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661437800.gz msg="downloaded file" total_time=5.357309ms level=info ts=2022-08-26T08:32:31.545982209Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661438700.gz msg="downloaded file" total_time=6.146379ms level=info ts=2022-08-26T08:32:31.547256105Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661439600.gz msg="downloaded file" total_time=7.35292ms level=info ts=2022-08-26T08:32:31.548053298Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661440500.gz msg="downloaded file" total_time=8.066054ms level=info ts=2022-08-26T08:32:31.549428783Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661441400.gz msg="downloaded file" total_time=9.391087ms level=info ts=2022-08-26T08:32:31.55027799Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661442300.gz msg="downloaded file" total_time=9.971864ms level=info ts=2022-08-26T08:32:31.550859318Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661444100.gz msg="downloaded file" total_time=10.428326ms level=info ts=2022-08-26T08:32:31.551707413Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661445000.gz msg="downloaded file" total_time=11.060339ms level=info ts=2022-08-26T08:32:31.566262924Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661443200.gz msg="downloaded file" total_time=25.910547ms level=info ts=2022-08-26T08:32:31.566893042Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661445900.gz msg="downloaded file" total_time=26.193913ms level=error ts=2022-08-26T08:32:31.569116703Z caller=index_set.go:265 table-name=index_19229 msg="sync failed, retrying it" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=info ts=2022-08-26T08:32:31.570578665Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661428800.gz msg="downloaded file" total_time=1.132928ms level=info ts=2022-08-26T08:32:31.571005416Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661433300.gz msg="downloaded file" total_time=1.160638ms level=info ts=2022-08-26T08:32:31.571076408Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661435100.gz msg="downloaded file" total_time=1.10758ms level=info ts=2022-08-26T08:32:31.571279457Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661431500.gz msg="downloaded file" total_time=1.584257ms level=info ts=2022-08-26T08:32:31.571545418Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661429700.gz msg="downloaded file" total_time=2.031318ms level=info ts=2022-08-26T08:32:31.571816665Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661432400.gz msg="downloaded file" total_time=2.097763ms level=info ts=2022-08-26T08:32:31.57193676Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661436900.gz msg="downloaded file" total_time=1.841745ms level=info ts=2022-08-26T08:32:31.572061104Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661434200.gz msg="downloaded file" total_time=2.147428ms level=info ts=2022-08-26T08:32:31.572073253Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661437800.gz msg="downloaded file" total_time=1.997301ms level=info ts=2022-08-26T08:32:31.572280647Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661428658.gz msg="downloaded file" total_time=2.853888ms level=info ts=2022-08-26T08:32:31.57235635Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661445900.gz msg="downloaded file" total_time=1.548907ms level=info ts=2022-08-26T08:32:31.572438999Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661440500.gz msg="downloaded file" total_time=2.122886ms level=info ts=2022-08-26T08:32:31.572213366Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661436000.gz msg="downloaded file" total_time=2.239392ms level=info ts=2022-08-26T08:32:31.572570132Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661441400.gz msg="downloaded file" total_time=2.139386ms level=info ts=2022-08-26T08:32:31.572812556Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661442300.gz msg="downloaded file" total_time=2.370753ms level=info ts=2022-08-26T08:32:31.572950315Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661443200.gz msg="downloaded file" total_time=2.429572ms level=info ts=2022-08-26T08:32:31.573005211Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661445000.gz msg="downloaded file" total_time=2.337528ms level=info ts=2022-08-26T08:32:31.572573165Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661430600.gz msg="downloaded file" total_time=3.013793ms level=info ts=2022-08-26T08:32:31.573352308Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661439600.gz msg="downloaded file" total_time=3.007624ms level=info ts=2022-08-26T08:32:31.57418152Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661438700.gz msg="downloaded file" total_time=3.974368ms level=info ts=2022-08-26T08:32:31.575073714Z caller=index_set.go:422 table-name=index_19229 file-name=loki-stack-0-1661428388368789724-1661444100.gz msg="downloaded file" total_time=4.501579ms level=error ts=2022-08-26T08:32:31.577027069Z caller=index_set.go:265 table-name=index_19229 msg="sync failed, retrying it" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" level=error ts=2022-08-26T08:32:31.577047846Z caller=index_set.go:104 table-name=index_19229 msg="failed to initialize table index_19229, cleaning it up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference" recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference error creating index client github.com/grafana/loki/pkg/storage.(*store).storeForPeriod /src/loki/pkg/storage/store.go:245 github.com/grafana/loki/pkg/storage.(*store).init /src/loki/pkg/storage/store.go:162 github.com/grafana/loki/pkg/storage.NewStore /src/loki/pkg/storage/store.go:145 github.com/grafana/loki/pkg/loki.(*Loki).initStore /src/loki/pkg/loki/modules.go:522 github.com/grafana/dskit/modules.(*Manager).initModule /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 github.com/grafana/loki/pkg/loki.(*Loki).Run /src/loki/pkg/loki/loki.go:341 main.main /src/loki/cmd/loki/main.go:105 runtime.main /usr/local/go/src/runtime/proc.go:255 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581 error initialising module: store github.com/grafana/dskit/modules.(*Manager).initModule /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122 github.com/grafana/dskit/modules.(*Manager).InitModuleServices /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92 github.com/grafana/loki/pkg/loki.(*Loki).Run /src/loki/pkg/loki/loki.go:341 main.main /src/loki/cmd/loki/main.go:105 runtime.main /usr/local/go/src/runtime/proc.go:255 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581 level=error ts=2022-08-26T08:32:31.577272807Z caller=log.go:103 msg="error running loki" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference\nerror creating index client\ngithub.com/grafana/loki/pkg/storage.(*store).storeForPeriod\n\t/src/loki/pkg/storage/store.go:245\ngithub.com/grafana/loki/pkg/storage.(*store).init\n\t/src/loki/pkg/storage/store.go:162\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:145\ngithub.com/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:522\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:341\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:341\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

loki.yaml:

`apiVersion: apps/v1 kind: StatefulSet metadata: annotations: meta.helm.sh/release-name: loki-stack meta.helm.sh/release-namespace: loki-stack creationTimestamp: "2022-08-24T10:44:03Z" generation: 8 labels: app: loki app.kubernetes.io/managed-by: Helm chart: loki-2.14.1 heritage: Helm release: loki-stack name: loki-stack namespace: loki-stack resourceVersion: "18437927" uid: 7a046ddf-b33d-49ae-9184-1fe7643ba0df spec: podManagementPolicy: OrderedReady replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: loki release: loki-stack serviceName: loki-stack-headless template: metadata: annotations: checksum/config: cf033c43b4c54b9f75e6fbecb21f31d8c612f4ad185a7a4e04d8d169c25fefb3 prometheus.io/port: http-metrics prometheus.io/scrape: "true" creationTimestamp: null labels: app: loki name: loki-stack release: loki-stack spec: affinity: {} containers: - args: - -config.file=/etc/loki/loki.yaml image: grafana/loki:latest imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: loki ports: - containerPort: 3100 name: http-metrics protocol: TCP - containerPort: 9095 name: grpc protocol: TCP - containerPort: 7946 name: memberlist-port protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: {} securityContext: readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /tmp name: tmp - mountPath: /etc/loki name: config - mountPath: /data name: storage dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 10001 runAsGroup: 10001 runAsNonRoot: true runAsUser: 10001 serviceAccount: loki-stack serviceAccountName: loki-stack terminationGracePeriodSeconds: 4800 volumes: - emptyDir: {} name: tmp - name: config secret: defaultMode: 420 secretName: loki-stack updateStrategy: type: RollingUpdate volumeClaimTemplates:

  • apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null name: storage spec: accessModes:
    • ReadWriteOnce resources: requests: storage: 50Gi storageClassName: infra-nfs-client volumeMode: Filesystem status: phase: Pending`

Note: I changed loki image tag from 2.6.5 to latest expecting it would resolve the issue but its the same neverthless.

loki-cm.yaml: apiVersion: v1 data: loki-stack-datasource.yaml: |- apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://loki-stack:3100 version: 1 isDefault: true kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: loki-stack meta.helm.sh/release-namespace: loki-stack creationTimestamp: "2022-08-24T10:44:02Z" labels: app: loki-stack app.kubernetes.io/managed-by: Helm chart: loki-stack-2.7.0 grafana_datasource: "1" heritage: Helm release: loki-stack name: loki-stack namespace: loki-stack resourceVersion: "17216171" uid: decb0b9a-87db-4182-a92c-82e0eb9014ba

saigha01 avatar Aug 26 '22 08:08 saigha01