outofsync error with StatefulSet

[lifeofmoo]

Hola,

EKS: v1.27 Argo: v2.8.4 EBS CSI Driver: v1.27.0-eksbuild.1

How do I fix this outofsync error please?

I've just changed my Deployment to a Stateful set.

---
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  # namespace: nextjs
  name: bp-knowledge-graph-api
spec:
  serviceAccountName: bp-knowledge-graph-api
  replicas: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: bp-knowledge-graph-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: bp-knowledge-graph-api
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - bp-knowledge-graph-api
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - eu-west-1c
      containers:
        - name: bp-knowledge-graph-api
          image: "111111111111111.dkr.ecr.eu-west-1.amazonaws.com/bp-knowledge-graph-api:xxxxxx"
          command: ["java"]
          imagePullPolicy: Always
          livenessProbe:
            initialDelaySeconds: 60
            periodSeconds: 30
            httpGet:
              path: /knowledge-graph-api/health-check
              port: 8080
          readinessProbe:
            initialDelaySeconds: 30
            httpGet:
              path: /knowledge-graph-api/health-check
              port: 8080
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              memory: 500Mi
              cpu: 500m
            limits:
              memory: 1Gi
          volumeMounts:
          - name: bp-graph-api-volume
            mountPath: /home/bmjdev
  volumeClaimTemplates:
    - metadata:
        name: bp-graph-api-volume
      spec:
        accessModes: [ "ReadWriteOncePod" ]
        resources:
          requests:
            storage: 35Gi
        storageClassName: gp3-retain
  # persistentVolumeClaimRetentionPolicy:
  #   whenDeleted: Retain
  #   whenScaled: Retain

I am not sure if the recently added volumeClaimTemplates and persistentVolumeClaimRetentionPolicy is causing this issue. I wasn't seeing this when I was using a PVC. However I changed it to volumeClaimTemplates to be more aligned with doing it the StatefulSet way.

Even commenting out the persistentVolumeClaimRetentionPolicy lines and re-creating the app results in an out of sync state.

Argo Logs look OK.

k logs argocd-application-controller-0 -n argocd | grep graph

> time="2024-04-25T11:16:07Z" level=info msg="Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2024-04-25 11:10:08 +0000 UTC, expiry: 3m0s), level (2)" application=argocd/bp-knowledge-graph-api
> time="2024-04-25T11:16:07Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: bp-knowledge-graph)" application=argocd/bp-knowledge-graph-api
> time="2024-04-25T11:16:08Z" level=info msg="getRepoObjs stats" application=argocd/bp-knowledge-graph-api build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=120 unmarshal_ms=120 version_ms=0
> time="2024-04-25T11:16:08Z" level=info msg="Skipping auto-sync: most recent sync already to 51b9ecac11456557306c1da26b738d55c58fa90a" application=argocd/bp-knowledge-graph-api
> time="2024-04-25T11:16:08Z" level=info msg="Update successful" application=argocd/bp-knowledge-graph-api
> time="2024-04-25T11:16:08Z" level=info msg="Reconciliation completed" application=argocd/bp-knowledge-graph-api dedup_ms=0 dest-name= dest-namespace=bp-knowledge-graph dest-server="https://kubernetes.default.svc" diff_ms=147 fields.level=2 git_ms=121 health_ms=49 live_ms=12 settings_ms=0 sync_ms=14 time_ms=446

k describe statefulsets.apps bp-knowledge-graph-api - looks ok!

Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  32m   statefulset-controller  create Claim bp-graph-api-volume-bp-knowledge-graph-api-0 Pod bp-knowledge-graph-api-0 in StatefulSet bp-knowledge-graph-api success
  Normal  SuccessfulCreate  32m   statefulset-controller  create Pod bp-knowledge-graph-api-0 in StatefulSet bp-knowledge-graph-api successful

[lifeofmoo]

Is the fix to annotate the App with argocd.argoproj.io/compare-options: ServerSideDiff=true ?

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bp-knowledge-graph-api
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  destination:
    namespace: bp-knowledge-graph
    server: https://kubernetes.default.svc
  project: bp-knowledge-graph
  source:
    path: bp-knowledge-graph-api/overlays/dev
    repoURL: https://github.com/ME-Ltd/my-repo
    targetRevision: HEAD
  syncPolicy:
    automated: {}

Because this didn't work for me.

[abhipsnl]

I tested this and went into panic mode, seems there is an issue created

https://github.com/argoproj/argo-cd/issues/18020