prune option doesn't play nice with CiliumIdentity resource

[michalschott]

Hey,

I'm running on EKS with Cilium as CNI, deploying apps with kustomize with prune option enabled.

Given my kustomization looks like this:

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: secrets-store-csi-driver
resources:
- aws-provider-installer.yaml
- namespace.yaml
- secrets-store-csi-driver.yaml

namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: secrets-store-csi-driver
  labels:
    goldilocks.fairwinds.com/enabled: "true"
  annotations:
    downscaler/exclude: "true"

Cilium automatically creates CiliumIdentity object, which ArgoCD obviously catch up and here is where story begins:

• app (especially CiliumIdentity) is constantly marked as Out-Of-Sync
• resource is being constantly removed by Argo
• Cilium recreates the object

How can I prevent this behaviour without turning off prune option?

[ashutosh16]

@michalschott You can add an annotation to the specific object CiliumIdentity, preventing it from pruning. Is this option work for you?

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#no-prune-resources

[michalschott]

@ashutosh16 not really, as these objects are not included in kustomization and are created by cilium itself

[debu99]

DigitalOcean Kubernetes are using cilium by default, i have the same issue

[michalschott]

@debu99

Apparently this fixed the issue for me:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      clusters:
      - "*"

Should be included as a default IMHO.

[sathieu]

Yes. This is documented here: https://docs.cilium.io/en/stable/configuration/argocd-issues/#argo-cd-deletes-customresourcedefinitions

Still, I don't understand why ArgoCD catches those resources. Maybe they copy the labels and annotations from the pods. Could this be a bug in how ArgoCD prunes?

[lebenitza]

Still, I don't understand why ArgoCD catches those resources. Maybe they copy the labels and annotations from the pods. Could this be a bug in how ArgoCD prunes?

apiVersion: cilium.io/v2
kind: CiliumIdentity
metadata:
  labels:
    app: webhook
    app.kubernetes.io/component: webhook
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: webhook
    app.kubernetes.io/version: v1.11.0
    helm.sh/chart: cert-manager-v1.11.0
    io.cilium.k8s.policy.cluster: default
    io.cilium.k8s.policy.serviceaccount: cert-manager-webhook
    io.kubernetes.pod.namespace: ingress-system

I think you are right, don't think it is a bug in ArgoCD. They need to be exluded.

[zyzyx03]

Glad I found this, Thanks everyone. The issue still persist in ArgoCD version 2.5.8

[jannfis]

To prevent this behaviour, you can set the resource tracking method to annotation, as described here. You will still see the resources created by Cilium, but Argo CD will not consider them out of sync or prune them.

[Kerwood]

@debu99

Apparently this fixed the issue for me:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      clusters:
      - "*"

Should be included as a default IMHO.

Is this still the prefered method of dealing with this issue ? Using ArgoCD v2.7.5.

[ofirshtrull]

@debu99 Apparently this fixed the issue for me:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      clusters:
      - "*"

Should be included as a default IMHO.

Is this still the prefered method of dealing with this issue ? Using ArgoCD v2.7.5.

@Kerwood FYI this still works for 2.8.2

[ebuildy]

These resources should belong to another existing resources right? And argocd handle well parent -> child relationship.

So it could be a bug from Cilium itself ( I mean Cilium should add a "ownerReferences" field) ?

[jgwest]

Confirmed, I see this on DigitalOcean K8s as well, and is likely a case where Cilium and Argo CD have incompatible behaviours.

By default, Argo CD uses app.kubernetes.io/instance label to keep track of whick K8s resources on the cluster are child resources of an Argo CD Application: https://argo-cd.readthedocs.io/en/stable/user-guide/resource_tracking/

• However, if another controller on the cluster adds this label to a resource (CiliumIdentity), and the label value matches an existing Argo CD Application name, then Argo CD will assume it owns that resource (and the resource will appear in Argo CD UI/CLI lists)

I can confirm, the workaround described above definitely works. This is the standard workaround for the case where another controller on the cluster is incompatible with Argo CD. You can also follow the instructions on this page, to use a custom label, or to annotation/annotation+label-based tracking, as Jann mentioned above.

Given that, I am closing this bug.