[loki-distributed] How to increase ingester disk space

[pinoma1]

Hi, I'm testing Loki for future use in our production Kubernetes cluster on AKS. I'm using the helm chart "loki-distributed". This is my configuration:

ingester:
  replicas: 1
  nodeSelector:
    kubernetes.io/os: linux
  persistence:
    enabled: true
    size: 2Gi
    storageClass: managed-csi-premium-retain # Same as managed-csi-premium but with retain policy

distributor:
  nodeSelector:
    kubernetes.io/os: linux
querier:
  nodeSelector:
    kubernetes.io/os: linux
queryFrontend:
  nodeSelector:
    kubernetes.io/os: linux
gateway:
  nodeSelector:
    kubernetes.io/os: linux

loki:
  structuredConfig:
    ingester:
      # Disable chunk transfer which is not possible with statefulsets
      # and unnecessary for boltdb-shipper
      max_transfer_retries: 0
      chunk_idle_period: 1h
      chunk_target_size: 1536000
      max_chunk_age: 1h
    storage_config:
      azure:
        account_key: xxx
        account_name: yyy
        container_name: zzz
        request_timeout: 0
      boltdb_shipper:
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 24h
        shared_store: azure
      filesystem:
        directory: /var/loki/chunks
    schema_config:
      configs:  
        - from: 2020-12-11
          store: boltdb-shipper
          index:
            period: 24h
            prefix: loki_index_
          object_store: azure
          schema: v11

How do you increase the disk space of the ingester when needed?

[trevorwhitney]

you would increase the values of ingester.persistence.size

[vitobotta]

Hi @trevorwhitney , I tried this now but get Error: UPGRADE FAILED: cannot patch "loki-ingester" with kind StatefulSet: StatefulSet.apps "loki-ingester" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Any suggestions?

[waney316]

anyone slove the problem ？

[zanhsieh]

Maybe try this?

https://serverfault.com/a/997236/319811