gitops-operator icon indicating copy to clipboard operation
gitops-operator copied to clipboard

Application-scoped Argo instance fails to sync when a managed namespace is deleted

Open etsauer opened this issue 4 years ago • 4 comments

Describe the bug When using an "Application-scoped" argocd (e.g. one that is not scoped to All Namepsaces in a cluster), the GitOps operator dynamically maintains a list of "managed namespaces", which are namespaces that include the argocd.argoproj.io/managed-by label. This list of namespaces gets cached in argocd (Settings > Clusters > "in-cluster" > NAMESPACES). However, if one of these namespaces gets deleted, the operator doesn't seem to notice, and the namespace remains in that list. This breaks both the "CONNECTION STATE" in that cluster config, and also all Sync attempts after the namespace gets deleted. The sync will show errors like <argo service account> does not have <random verb> on <random resource type> in namespace "the-namespace-you-deleted".

To Reproduce Steps to reproduce the behavior:

  1. Deply an application scoped ArgoCD via the CR:

    apiVersion: argoproj.io/v1alpha1
    kind: ArgoCD
    metadata:
     name: argocd #name of the Argo CD instance
     namespace: foo #namespace where you want to deploy argocd instance
    spec:
     server:
       route:
         enabled: true #creates an openshift route to access Argo CD UI
    
  2. Create two new namespaces and label them

    kubectl create namespace to-delete
    kubectl label namespace/to-delete argocd.argoproj.io/managed-by=foo
    kubectl create namesoace sample-app
    kubectl label namespace/sample-app argocd.argoproj.io/managed-by=foo
    
  3. Delete the first namespace

    kubectl delete namepsace/to-delete
    
  4. Apply an application to second namespace

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: sample-app #app CR name
      namespace: foo #argocd instance namespace
    spec:
      destination:
        namespace: sample-app #namespace where app is deployed
        server: 'https://kubernetes.default.svc'
      source:
        path: app
        repoURL: 'https://github.com/redhat-developer/openshift-gitops-getting-started'
        targetRevision: HEAD
      project: default
    
  5. Observe the following errors:

  6. Continual sync errors under App Conditions for the sample app, complaining of permissions errors for the to-delete namespace

  7. Connection errors under the cluster settings

Expected behavior The namespace should be deleted from the list of manages namespaces in the cluster settings, and the app should sync successfully.

etsauer avatar Sep 28 '21 13:09 etsauer

I think @shubhamagarwal19 can help us on this :)

iam-veeramalla avatar Sep 28 '21 14:09 iam-veeramalla

Btw.. I have observed this behavior in both v1.2.0 and v1.2.1 of the operater (same instance before and after an upgrade)

etsauer avatar Sep 28 '21 14:09 etsauer

https://issues.redhat.com/browse/GITOPS-1521

nccurry avatar Nov 07 '21 21:11 nccurry

Upstream issue: https://github.com/argoproj-labs/argocd-operator/issues/478

Upstream PR: https://github.com/argoproj-labs/argocd-operator/pull/479

jaideepr97 avatar Nov 09 '21 21:11 jaideepr97