loki icon indicating copy to clipboard operation
loki copied to clipboard

Storage resizing doesn't work with SingleBinary deployment.

Open mindthecap opened this issue 9 months ago • 1 comments

Storage resize in SingleBinary deployment is impossible because it's created with StatefulSets, which prohibits changing the manifest after deployment. The affected line in code: https://github.com/grafana/loki/blob/599a3002d2878eafd2a64dc270f662f9756cae5b/production/helm/loki/templates/single-binary/statefulset.yaml#L172-L191

SingleBinary deployment should create separate PVC and reference it in the StatefulSet.

mindthecap avatar May 02 '24 09:05 mindthecap

A workaround for this:

  1. Resize the PVC and PV.
  2. Delete the statefulset but keep the pod kubectl delete sts --cascade=orphan loki.
  3. Redeploy Loki with helm.
  4. Delete the pod to trigger volume resizing.

More info in here: https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set

mindthecap avatar May 02 '24 10:05 mindthecap