prune option doesn't play nice with CiliumIdentity resource

[michalschott]

Hey,

I'm running on EKS with Cilium as CNI, deploying apps with kustomize with prune option enabled.

Given my kustomization looks like this:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: secrets-store-csi-driver
resources:
- aws-provider-installer.yaml
- namespace.yaml
- secrets-store-csi-driver.yaml

namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: secrets-store-csi-driver
  labels:
    goldilocks.fairwinds.com/enabled: "true"
  annotations:
    downscaler/exclude: "true"

Cilium automatically creates CiliumIdentity object, which ArgoCD obviously catch up and here is where story begins:

• app (especially CiliumIdentity) is constantly marked as Out-Of-Sync
• resource is being constantly removed by Argo
• Cilium recreates the object

How can I prevent this behaviour without turning off prune option?

[ashutosh16]

@michalschott You can add an annotation to the specific object CiliumIdentity, preventing it from pruning. Is this option work for you?

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#no-prune-resources

[michalschott]

@ashutosh16 not really, as these objects are not included in kustomization and are created by cilium itself

[debu99]

DigitalOcean Kubernetes are using cilium by default, i have the same issue

[michalschott]

@debu99

Apparently this fixed the issue for me:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      clusters:
      - "*"

Should be included as a default IMHO.

[sathieu]

Yes. This is documented here: https://docs.cilium.io/en/stable/configuration/argocd-issues/#argo-cd-deletes-customresourcedefinitions

Still, I don't understand why ArgoCD catches those resources. Maybe they copy the labels and annotations from the pods. Could this be a bug in how ArgoCD prunes?