Simple Scalable loki pods are in CrashLoopBackOff.

[sunidhi271]

Describe the bug Simple Scalable loki pods are in CrashLoopBackOff.

To Reproduce Steps to reproduce the behavior:

• Create a proxy helm chart for loki and install that with the value file configurations below.

1. helm repo add grafana https://grafana.github.io/helm-charts
2. helm repo update

create a proxy helm chart

3. helm create loki
4. Update the chart.yaml with the following -

apiVersion: v2
name: loki-promtail
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"

dependencies:
#loki version 3.0.0
- name: loki
  version: "6.5.2"
  repository: "https://grafana.github.io/helm-charts"

5. update the helm chart with loki chart helm dependency update .

6. Update the value file with the below values: cat values.yaml

##Default value file
loki:
  global:
    image:
      registry: registry.xyz.com
    fullnameOverride: loki
    imagePullSecrets: [dacsecret]
  test:
    enabled: false
  gateway:
    enabled: false
  lokiCanary:
    enabled: false
  monitoring:
    selfMonitoring:
      grafanaAgent:
        installOperator: false

# SimpleScalable Mode related values
  deploymentMode: SimpleScalable
  sidecar:
    image:
      repository: registry.xyz.com/public/kiwigrid/k8s-sidecar
      tag: 1.24.3
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    rules:
      enabled: true
      label: loki_rule
      labelValue: ""
      folder: /rules
  memcached:
    image:
      repository: registry.xyz.com/public/memcached
#default#      tag: 1.6.23-alpine 
      tag: 1.6.25
  memcachedExporter:
    image: 
      repository: registry.xyz.com/public/prom/memcached-exporter
      tag: v0.14.2
  minio:
    enabled: true
    image:
      repository: registry.xyz.com/public/minio
    mcImage:
      repository: registry.xyz.com/public/quay.io/minio/mc
  backend:
    replicas: 3
    autoscaling:
      enabled: false
      minReplicas: 3
      maxReplicas: 6
    persistence:
      volumeClaimsEnabled: true
      # -- Parameters used for the `data` volume when volumeClaimEnabled if false
      dataVolumeParameters:
        emptyDir: {}
      # -- Enable StatefulSetAutoDeletePVC feature
      enableStatefulSetAutoDeletePVC: false
      size: 10Gi
      storageClass: "rook-block"
      # -- Selector for persistent disk
      selector: null
    resources:
      limits:
        memory: 50Gi
      requests:
        memory: 1Gi
  read:
    replicas: 3
    autoscaling:
      enabled: false
      minReplicas: 3
      maxReplicas: 6
      targetCPUUtilizationPercentage: 60
    persistence:
      volumeClaimsEnabled: true
      size: 10Gi
      storageClass: rook-block
    resources:
      limits:
        memory: 50Gi
      requests:
        memory: 1Gi
  write:
    replicas: 3
    autoscaling:
      enabled: false
      minReplicas: 3
      maxReplicas: 6
      targetCPUUtilizationPercentage: 60
      resources: {}
    persistence:
      volumeClaimsEnabled: true
      # -- Parameters used for the `data` volume when volumeClaimEnabled if false
      dataVolumeParameters:
        emptyDir: {}
      enableStatefulSetAutoDeletePVC: false
      size: 10Gi
      storageClass: "rook-block"
      selector: null
    resources:
      limits:
        memory: 50Gi
      requests:
        memory: 1Gi
  tableManager:
    enabled: false
    extraVolumes:
      - name: data
        emptyDir: {}
    extraVolumeMounts:
      - name: data
        mountPath: /var/loki
    retention_period: 24h

  loki:
    image:
      registry: registry.xyz.com
      repository: public/grafana/loki
    ingester:
      chunk_encoding: snappy
      replicas: 0
    tracing:
      enabled: true
    querier:
      # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
      max_concurrent: 4
      replicas: 0
    minio:
      enabled: true
    singleBinary:
      replicas: 0
    queryFrontend:
      replicas: 0
    queryScheduler:
      replicas: 0
    distributor:
      replicas: 0
    compactor:
      replicas: 0
    indexGateway:
      replicas: 0
    bloomCompactor:
      replicas: 0
    bloomGateway:
      replicas: 0
#SimpleScalable Mode related values ends here

#    structuredConfig:
    config: |
      auth_enabled: false
      limits_config:
        ingestion_rate_strategy: local
        reject_old_samples: true
        reject_old_samples_max_age: 168h
        ingestion_rate_mb: 400
        ingestion_burst_size_mb: 600
        max_global_streams_per_user: 10000
        max_query_length: 72h
        max_query_parallelism: 64
        cardinality_limit: 200000
        split_queries_by_interval: 30m
    schemaConfig:
      configs:
        - from: 2024-04-01
          object_store: s3
          store: tsdb
          schema: v13
          index:
            prefix: index_
            period: 24h
    auth_enabled: false
    commonConfig:
      replication_factor: 1

7. Take out the templates and deploy it to kubernetes cluster: helm template --name-template=loki --namespace=loki . > out.yaml kubectl apply -f out.yaml

Expected behavior All pods should be running state and minio pvc should have got mounted. But below is the status:

Environment:

• Infrastructure: Kubernetes {v1.21.9}
• Deployment mode: Manual

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.

[sunidhi271]

@elchenberg adthonb JStickler TsengSR alexandreMegel Could any of you help here ?

[sunidhi271]

With this value file configuration at least the pods are now in running state:

###Promtail Values###
promtail:
  fullnameOverride: promtail
  image:
    registry: registry.dac.nokia.com
    repository: public/grafana/promtail
  config:
    clients:
      - url: http://loki-gateway/loki/api/v1/push
  extraPorts:
    syslog:
      name: promtail
      annotations: {}
      labels: {}
      containerPort: 8514
      protocol: TCP
      service:
          type: ClusterIP
          clusterIP: null
          port: 8514
          externalIPs: []
          nodePort: null
          loadBalancerIP: null
          loadBalancerSourceRanges: []
          externalTrafficPolicy: null

#########Loki Values #########
##Default value file
loki:
  global:
    image:
      registry: registry.xyz.com
    fullnameOverride: loki
    imagePullSecrets: [dacsecret]
  test:
    enabled: false
  gateway:
    enabled: true
    image:
      registry: registry.xyz.com
      repository: public/nginxinc/nginx-unprivileged
      tag: 1.24-alpine
  lokiCanary:
    enabled: false
  chunksCache:
    enabled: false
  resultsCache:
    enabled: false

# SimpleScalable Mode related values
  deploymentMode: SimpleScalable
  sidecar:
    image:
      repository: registry.xyz.com/public/kiwigrid/k8s-sidecar
      tag: 1.24.3
  memcached:
    image:
      repository: registry.xyz.com/public/memcached
#default#      tag: 1.6.23-alpine 
      tag: 1.6.25
  memcachedExporter:
    image: 
      repository: registry.xyz.com/public/prom/memcached-exporter
      tag: v0.14.2
  minio:
    enabled: true
    image:
      repository: registry.xyz.com/public/minio
    mcImage:
      repository: registry.xyz.com/public/quay.io/minio/mc
    persistence:
      enabled: true
      storageClass: "rook-block"
      size: 20Gi
  backend:
    replicas: 2
    persistence:
      volumeClaimsEnabled: true
      size: 10Gi
      storageClass: "rook-block"
      enableStatefulSetAutoDeletePVC: false
  read:
    replicas: 2
    persistence:
      volumeClaimsEnabled: true
      storageClass: "rook-block"
      enableStatefulSetAutoDeletePVC: false
  write:
    replicas: 2
    persistence:
      volumeClaimsEnabled: true
      storageClass: "rook-block"
    enableStatefulSetAutoDeletePVC: false
  singleBinary:
    replicas: 0
  ingester:
    replicas: 0
  querier:
    replicas: 0
  queryFrontend:
    replicas: 0
  queryScheduler:
    replicas: 0
  distributor:
    replicas: 0
  compactor:
    replicas: 0
  indexGateway:
    replicas: 0
  bloomCompactor:
    replicas: 0
  bloomGateway:
    replicas: 0
  tableManager:
    enabled: false


#    extraArgs:
#      - -config.expand-env=true
#    extraEnv:
#      - name: GRAFANA-LOKI-S3-ENDPOINT
#        valueFrom:
#          secretKeyRef:
#            name: loki-credentials
#            key: s3-endpoint
#      - name: GRAFANA-LOKI-S3-ACCESSKEY
#        valueFrom:
#          secretKeyRef:
#            name: loki-credentials
#            key: s3-access-key
#      - name: GRAFANA-LOKI-S3-SECRETKEY
#        valueFrom:
#          secretKeyRef:
#            name: loki-credentials
#            key: s3-secret-key
#    extraEnvFrom:
#      - secretRef:
#          name: loki-s3-secret

  loki:
    image:
      registry: registry.dac.nokia.com
      repository: public/grafana/loki
    schemaConfig:
      configs:
        - from: 2024-04-01
          store: tsdb
          object_store: s3
          schema: v13
          index:
            prefix: loki_index_
            period: 24h
    ingester:
      chunk_encoding: snappy
    tracing:
      enabled: true
    querier:
      # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
      max_concurrent: 4
    minio:
      enabled: true
    auth_enabled: false