ServerSideDiff not detecting diff in manifests

[klaudworks]

Describe the bug

If server side diff is enabled, Argo sometimes does not detect diffs when fields are added to a resource or removed from a resource.

To Reproduce

Note: To really replicate the issue, you would have to install the Strimzi Kafka Operator. I encountered this error before and will update this issue once I encounter an easier to reproduce example.

1. Enable server side diff via Annotation: argocd.argoproj.io/compare-options=ServerSideDiff=true,IncludeMutationWebhook=true.

2. Apply KafkaConnector Resource via ArgoCD:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  labels:
    strimzi.io/cluster: backup-connect-cluster
  name: s3-sink-consumer-offsets
  namespace: mynamespace
spec:
  autoRestart:
    enabled: true
  class: io.lenses.streamreactor.connect.aws.s3.sink.S3ConsumerGroupsSinkConnector
  tasksMax: 1

3. Remove the field autoRestart via git. Then, Argo shows the live manifest correctly with autoRestart.enabled=true and it shows the desired manifest correctly with autoRestart field missing. However, there is no Diff shown. Turning off server side diff fixes the issue.

Expected behavior

Argo detecting diffs properly with server side diff.

Version

Argo CD Version: v3.0.0+e98f483 Kubernetes Versions: 1.32.0 and 1.32.2

[klaudworks]

#23066 doesn't fix this because the problem also occurs when deploying the resource without an autoRestart field and then adding it.

[pjiang-dev]

@klaudworks Do you mind showing the complete resource with its managed fields? My guess is the operator manages the fields instead of argo cd.

[klaudworks]

Here is another example I just encountered with an ingress resource:

Or here I removed the rewrite annotation nginx.ingress.kubernetes.io/rewrite-target: /$2 from the ingress: but it is not recognized:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: >-
      stage-frontend-admin:networking.k8s.io/Ingress:XXX
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  creationTimestamp: '2025-04-22T13:57:33Z'
  generation: 4
  labels:
    app.kubernetes.io/instance: frontend
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: frontend
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: frontend-0.1.0
  managedFields:
    - apiVersion: networking.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:argocd.argoproj.io/tracking-id: {}
            f:cert-manager.io/cluster-issuer: {}
            f:nginx.ingress.kubernetes.io/proxy-body-size: {}
            f:nginx.ingress.kubernetes.io/rewrite-target: {}
          f:labels:
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:ingressClassName: {}
          f:rules: {}
          f:tls: {}
      manager: argocd-controller
      operation: Apply
      time: '2025-06-04T14:59:55Z'
    - apiVersion: networking.k8s.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      manager: nginx-ingress-controller
      operation: Update
      subresource: status
      time: '2025-04-22T14:06:04Z'
  name: frontend
  namespace: kimi
  resourceVersion: '36621983867'
  uid: b2b77089-2cba-4a0b-95cb-5c939c9a7a1e
spec:
  ingressClassName: nginx
  rules:
    - host: XXX
      http:
        paths:
          - backend:
              service:
                name: frontend
                port:
                  number: 80
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - frontend.example.com
      secretName: frontend.example.com-tls
status:
  loadBalancer:
    ingress:
      - ip: XXX

[pjiang-dev]

@klaudworks Looking at your example, the field you are describing that doesn't show a diff is indeed owned by another controller/operator

            f:ingress: {}
      manager: nginx-ingress-controller

This means argocd will not own this field and will not track changes if there is a diff with the field. If you do not expect the controller to modify these fields anymore and would like argo-cd to own these fields and show a diff you can use this new feature https://argo-cd.readthedocs.io/en/latest/user-guide/sync-options/#client-side-apply-migration

In your case you would set this annotation and use Server-Side Apply:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/client-side-apply-migration-manager: "nginx-ingress-controller"

This would allow argo-cd to manage the 'ingress' field and show diffs.