loki icon indicating copy to clipboard operation
loki copied to clipboard

Fix PVC in loki helm

Open cbcoutinho opened this issue 1 year ago • 4 comments

The lacking apiVersion and kind attributes make ArgoCD go into an endless loop

What this PR does / why we need it:

The PVC lacks attributes which are applied by Kubernetes; however, this leaves ArgoCD in a loop because the attributes are different.

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Checklist

  • [ ] Reviewed the CONTRIBUTING.md guide (required)
  • [ ] Documentation added
  • [ ] Tests updated
  • [ ] Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • [ ] For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • [ ] If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

cbcoutinho avatar Sep 10 '24 15:09 cbcoutinho

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Sep 10 '24 15:09 CLAassistant

This is the diff that ArgoCD detects when I use PVCs for the loki-chunks-cache StatefulSet - it tries to prune the fields that ArgoCD generates from the helm chart.

The only workaround I've found is to use the following stanza in my Application manifest for ArgoCD, ignoring those fields:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: observability
  namespace: argocd
spec:
  project: ...
  ignoreDifferences:
    - group: apps
      kind: StatefulSet
      name: observability-loki-chunks-cache
      jqPathExpressions:
        - .spec.volumeClaimTemplates[].apiVersion
        - .spec.volumeClaimTemplates[].kind
  sources:
    - repoURL: https://grafana.github.io/helm-charts
      chart: loki
      targetRevision: 6.12.0

The persistence is enabled for the loki-chunks-cache by selecting the storage class in the values.yml file

chunksCache:
  # Persistence settings for the chunks-cache
  persistence:
    enabled: true # TODO: Endless loop
    storageClass: nfs-client

Screenshot 2024-09-14 at 13 06 00

cbcoutinho avatar Sep 14 '24 11:09 cbcoutinho

Hi @chaudum, I confirmed that al the persistent volume claims required by loki components appear to have this issue, so I currently have to add the ignoreDifferences block for each StatefulSet deployed via helm. For the singleBinarry installation with caching enabled, this requires the initial loki volume, as well as volumes for the chunks-cache and results-cache

cbcoutinho avatar Sep 18 '24 08:09 cbcoutinho

https://github.com/grafana/loki/pull/15192 should have fixed it.

aslafy-z avatar Jun 12 '25 14:06 aslafy-z

Yes, this can be closed now.

Starefossen avatar Jun 24 '25 14:06 Starefossen

Closing, fixed via https://github.com/grafana/loki/pull/15192. Thank you for taking the time to contribute to the Loki project.

JStickler avatar Jun 25 '25 20:06 JStickler