can't delete resource - CR stuck

[davidkarlsen]

I tried deleting a NamespaceConfiguration resource, but it won't be deleted due to a stuck finalizer, and the operator tries updating the object with a finalizer, which is not allowed because it is being deleted

2021-09-09T14:53:26.583Z        ERROR   controllers.NamespaceConfig     unable to update instance       {"namespaceconfig": "/fss-apps", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "NamespaceConfig", "name": "fss-apps"}, "error": "NamespaceConfig.redhatcop.redhat.io \"fss-apps\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"namespaceconfig-controller\"}"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile
        /workspace/controllers/namespaceconfig_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99

[raffaelespazzoli]

can you paste the instance(yaml)? There should always ever be one finalizer. And the logic should be: if the instance is new, the finalizer is added, if the instance is being delete and the clean up has been completed the finalizer is removed...

[davidkarlsen]

--- a/manifests/governance/namespace-configurator/rolebindings.yaml
+++ /dev/null
@@ -1,44 +0,0 @@
-apiVersion: redhatcop.redhat.io/v1alpha1
-kind: NamespaceConfig
-metadata:
-  name: fss-apps
-spec:
-  labelSelector:
-    matchLabels:
-      app.kubernetes.io/managed-by: clout
-      fss.tietoevry.com/legacy-logging: "true"
-  templates:
-  - objectTemplate: |
-      apiVersion: rbac.authorization.k8s.io/v1
-      kind: RoleBinding
-      metadata:
-        name: fss-apps
-        namespace: {{ .Name }}
-        labels:
-          app.kubernetes.io/managed-by: namespace-configuration-operator
-      roleRef:
-        apiGroup: rbac.authorization.k8s.io
-        kind: ClusterRole
-        name: fss-apps
-      subjects:
-      - apiGroup: rbac.authorization.k8s.io
-        kind: Group
-        name: system:serviceaccounts:{{ .Name }}

[raffaelespazzoli]

sorry I meant the resource how it appears in the cluster with all of the fields.

On Fri, Sep 10, 2021 at 2:27 PM David J. M. Karlsen < @.***> wrote:

--- a/manifests/governance/namespace-configurator/rolebindings.yaml +++ /dev/null @@ -1,44 +0,0 @@ -apiVersion: redhatcop.redhat.io/v1alpha1 -kind http://redhatcop.redhat.io/v1alpha1-kind: NamespaceConfig -metadata:

• name: fss-apps -spec:
• labelSelector:
• matchLabels:

•  app.kubernetes.io/managed-by: clout
  

•  fss.tietoevry.com/legacy-logging: "true"
  

• templates:
• • objectTemplate: |

•  apiVersion: rbac.authorization.k8s.io/v1
  

•  kind: RoleBinding
  

•  metadata:
  

•    name: fss-apps
  

•    namespace: {{ .Name }}
  

•    labels:
  

•      app.kubernetes.io/managed-by: namespace-configuration-operator
  

•  roleRef:
  

•    apiGroup: rbac.authorization.k8s.io
  

•    kind: ClusterRole
  

•    name: fss-apps
  

•  subjects:
  

•  - apiGroup: rbac.authorization.k8s.io
  

•    kind: Group
  

•    name: system:serviceaccounts:{{ .Name }}
  

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-917119412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXE2WIV7HAAXHJ34D6TUBJE25ANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele

[davidkarlsen]

sorry, that's gone by now. I'll reopen a case if I can reproduce.

[davidkarlsen]

@raffaelespazzoli found the cause, the finalizers seems to have changed name at some point, leading to having two:

finalizers:
  - namespace-config-operator
  - namespaceconfig-controller

[raffaelespazzoli]

this should be the right one: "namespaceconfig-controller". Also can you share between which version you see the change, for the benefit of others. And sorry for the inconvenience.

On Tue, Oct 19, 2021 at 2:54 PM David J. M. Karlsen < @.***> wrote:

@raffaelespazzoli https://github.com/raffaelespazzoli found the cause, the finalizers seems to have changed name at some point, leading to having two:

finalizers:

• namespace-config-operator
• namespaceconfig-controller

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-947014976, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXADNOBC5DJICEHW2LTUHW5EVANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele

[davidkarlsen]

this should be the right one: "namespaceconfig-controller". Also can you share between which version you see the change, for the benefit of others. And sorry for the inconvenience. … On Tue, Oct 19, 2021 at 2:54 PM David J. M. Karlsen < @.***> wrote: @raffaelespazzoli <@raffaelespazzoli> found the cause, the finalizers seems to have changed name at some point, leading to having two: finalizers: - namespace-config-operator - namespaceconfig-controller — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <github.com/notifications/unsubscribe-auth/ABPERXADNOBC5DJICEHW2LTUHW5EVANCNFSM5DXLARPQ> . Triage notifications on the go with GitHub Mobile for iOS <apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. -- ciao/bye Raffaele

I don't know in which version, I've upgraded several times.

[davidkarlsen]

Maybe a fix could be made in the controller to look for the invalid one and simply remove it?

[raffaelespazzoli]

one can simply create a little script that removes the old annotations.

On Mon, Oct 25, 2021 at 4:31 AM David J. M. Karlsen < @.***> wrote:

Maybe a fix could be made to look for the invalid one and simply remove it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-950663748, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXHVJUK67HXAS7FWIXLUIUIW5ANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele