GKE 1.27: ComparisonError for CSIDriver .spec.seLinuxMount: field not declared in schema

[kaiburjack]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

After upgrading to Kubernetes/GKE 1.27, we get a sync error from Argo CD v2.7.2+cbee7e6 that a field (which became alpha in Kubernetes 1.25 but apparently now is stable/graduated in Kubernetes 1.27 and being set by the API server after applying a manifest which did not have this field in it) is not recognized:

ComparisonError: error calculating structured merge diff: error building typed value from live resource: .spec.seLinuxMount: field not declared in schema.

An ignoreDifferences on the respective Argo CD Application does solve this issue:

ignoreDifferences:
- group: storage.k8s.io
  jsonPointers:
  - /spec/seLinuxMount
  kind: CSIDriver
  name: nfs.csi.k8s.io

However it would be great if that wasn't needed.

To Reproduce

Apply the following CSIDriver manifest to a 1.27 cluster:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  labels:
    argocd.argoproj.io/instance: csi-driver-nfs
  name: nfs.csi.k8s.io
spec:
  attachRequired: false
  fsGroupPolicy: File
  volumeLifecycleModes:
    - Persistent

the live state of this resource would be:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  creationTimestamp: '2023-03-30T16:20:58Z'
  labels:
    argocd.argoproj.io/instance: csi-driver-nfs
  name: nfs.csi.k8s.io
  resourceVersion: '19449537'
  uid: 8038c804-bb65-41ec-aa4f-9437847f7682
spec:
  attachRequired: false
  fsGroupPolicy: File
  podInfoOnMount: false
  requiresRepublish: false
  seLinuxMount: false
  storageCapacity: false
  volumeLifecycleModes:
    - Persistent

Expected behavior

The structured YAML diff of Argo CD to work out that there is no difference between the live state and the desired state.

Version

v2.7.2+cbee7e6

[JoseAlvarezSonos]

Seeing the same issue with ArgoCD 2.7.3+e7891b8 on EKS 1.27.

[dels78]

Seeing the same with v2.7.3+e7891b8 on k3s v1.27.2+k3s1

[davhdavh]

same issue with kubernetes 1.28.2 and v2.8.2+dbdfc71

[noroutine]

same on vanilla 1.28.2 and v2.8.4+c279299

this can be mitigated by disabling feature gate SELinuxMountReadWriteOncePod (enabled by default in 1.28)

[dfroberg]

Same for EKS v1.28.1-eks

[monester]

Also have the same problem on v1.28.2+k3s1 and Argo CDv2.8.4+c279299 with latest synology-csi.

Removed warning by adding to Application:

  ignoreDifferences:
    - group: storage.k8s.io
      kind: CSIDriver
      jqPathExpressions:
        - .spec.seLinuxMount

[Aur0nd]

Still an issue.. But a great temporary solution is what @monester suggested.

v1.28.4 on prem k8s v2.0.3+6eba5be argo

[jtcressy]

Still experiencing this issue on argocd v2.9.3+6eba5be. K8S version v1.28.2, talos linux v1.5.5, running bare metal. Attempted to use ignoreDifferences on the application but the diff still fails and the application remains in Unknown state.

---

Edit: It seems the only way to make this go away was to disable the feature gate on my apiserver via SELinuxMountReadWriteOncePod=false and remove the seLinuxMount field from my CSIDriver object. Most managed k8s distributions won't be able to customize apiserver feature gates like this, so YMMV

[adberger]

Still experiencing this issue on argocd v2.9.3+6eba5be. K8S version v1.28.2, talos linux v1.5.5, running bare metal. Attempted to use ignoreDifferences on the application but the diff still fails and the application remains in Unknown state.

Edit: It seems the only way to make this go away was to disable the feature gate on my apiserver via SELinuxMountReadWriteOncePod=false and remove the seLinuxMount field from my CSIDriver object. Most managed k8s distributions won't be able to customize apiserver feature gates like this, so YMMV

Does anyone know, why the following doesn't work?

Argo Application

  ...
  ignoreDifferences:
  - group: storage.k8s.io
    jsonPointers:
    - /spec/seLinuxMount
    kind: CSIDriver

CSIDriver

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: testing_nutanix-csi-storage:storage.k8s.io/CSIDriver:nutanix-system/csi.nutanix.com
  creationTimestamp: "2023-12-28T08:35:36Z"
  labels:
    argocd.argoproj.io/instance: testing_nutanix-csi-storage
  name: csi.nutanix.com
  resourceVersion: "83290342"
  uid: a26eab8c-7373-4625-ab53-90d02b0e104a
spec:
  attachRequired: false
  fsGroupPolicy: ReadWriteOnceWithFSType
  podInfoOnMount: true
  requiresRepublish: false
  seLinuxMount: false
  storageCapacity: false
  volumeLifecycleModes:
  - Persistent

Problem occurs on ArgoCD v2.8.4 & v2.9.3

[algo7]

Confirmed same problem on ArgoCD v2.9.3+6eba5be with the official kube csi-driver-nfs v4.5.0

[myoung34]

Also seeing this in Argo 2.10 with secrets-store-csi-driver helm 1.4.1

I tried

      secrets-store.csi.k8s.io/csidriver.storage.k8s.io:
          ignoreDifferences: |
             jsonPointers:
             - /spec/seLinuxMount

and

      secrets-store.csi.k8s.io/csidriver
          ignoreDifferences: |
             jsonPointers:
             - /spec/seLinuxMount

neither worked, errors in argo

[kaiburjack]

@myoung34 The ignoreDifferences spec field of the ArgoCD Application is not meant to contain a YAML multiline string literal, but a structured YAML array/list. The workaround still works for me with latest Argo CD.

[myoung34]

@kaiburjack I configure it this way

[myoung34]

For those coming here:

It seems as though jsonPointers depends on the schema and errors, while jqPathExpressions does not

I tried

      storage.k8s.io/CSIDriver:
          ignoreDifferences: |
             jsonPointers :
              - /spec/seLinuxMount

but it did not work, while

      storage.k8s.io/CSIDriver:
          ignoreDifferences: |
             jqPathExpressions:
              - .spec.seLinuxMount

did in fact work and its now green and healthy

[woehrl01]

If you still encounter that issue, try to remove "managedFieldsManagers" in any of the matching "ignoreDiff" rules. It looks like that this happens if you have managedFieldsManagers set and the crd of the core resources in argocd does not match with the target cluster version.

[wpi86]

If you still encounter that issue, try to remove "managedFieldsManagers" in any of the matching "ignoreDiff" rules. It looks like that this happens if you have managedFieldsManagers set and the crd of the core resources in argocd does not match with the target cluster version.

Confirm, had similar issue, but with Jobs (error building typed results: error creating typedLive: errors: .spec.podReplacementPolicy: field not declared in schema). managedFieldsManagers was set on global level, after removing it ignoredifference start working. JobPodReplacementPolicy featuregate now in beta and enabled by default. eks 1.29 + argo v2.10.0+2175939

[laiminhtrung1997]

If you still encounter that issue, try to remove "managedFieldsManagers" in any of the matching "ignoreDiff" rules. It looks like that this happens if you have managedFieldsManagers set and the crd of the core resources in argocd does not match with the target cluster version.

Where is the "managedFieldsManagers"? I couldn't find it.

[wpi86]

If you still encounter that issue, try to remove "managedFieldsManagers" in any of the matching "ignoreDiff" rules. It looks like that this happens if you have managedFieldsManagers set and the crd of the core resources in argocd does not match with the target cluster version.

Where is the "managedFieldsManagers"? I couldn't find it.

https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/

[laiminhtrung1997]

Dear @wpi86 Could you please give me an example? Thanks a lot.

[nicholass-alcidion]

I am currenlty expericing this issue and do not have managedFieldsManagers anywhere in my repo - nor can i see it on the effected deployed application(s)

I have also checked the deployed argocd-cm and can't see this in that also.

is this possible to have had set at some global level via some method other than commited config?

[crenshaw-dev]

I'm able to reproduce this on the latest commit (dafb37c88d677527022ed8c8a7cfc9e875853d73) with Kubernetes 1.30.1.

I haven't set any ignoreDifferences. Instead I'm using ServerSideApply and then editing the podInfoOnMount to force the app out of sync. Then I consistently get the error.

If I don't edit the resource to force the app out of sync, I get the error as a blip, but it disappears the moment the app is finished syncing.

[crenshaw-dev]

Local test indicates that this will fix it: https://github.com/argoproj/gitops-engine/pull/588