community Error when deleting multiple resources using `foreground` cascading strategy

Follow up to https://github.com/aws-controllers-k8s/community/issues/1707

It looks like when delete multiple resources (+100) using foreground cascading strategy (kubectl delete queue --cascade=foreground --all) the controllers (sometimes) fail with the follow error:

2023-04-03T16:05:19.985+0200	DEBUG	ackrt	<<<< kc.Patch (metadata + spec)	{"account": "771174509839", "role": "", "region": "us-west-2", "kind": "Queue", "namespace": "default", "name": "ackqueue-0", "generation": 2, "error": "Queue.sqs.services.k8s.aws \"ackqueue-0\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"foregroundDeletion\"}"}
2023-04-03T16:05:19.985+0200	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"account": "771174509839", "role": "", "region": "us-west-2", "kind": "Queue", "namespace": "default", "name": "ackqueue-0", "generation": 2, "error": "Queue.sqs.services.k8s.aws \"ackqueue-0\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"foregroundDeletion\"}"}
2023-04-03T16:05:19.985+0200	DEBUG	ackrt	<< r.setResourceUnmanaged	{"account": "771174509839", "role": "", "region": "us-west-2", "kind": "Queue", "namespace": "default", "name": "ackqueue-0", "generation": 2, "error": "Queue.sqs.services.k8s.aws \"ackqueue-0\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"foregroundDeletion\"}"}
2023-04-03T16:05:19.985+0200	DEBUG	ackrt	< r.deleteResource	{"account": "771174509839", "role": "", "region": "us-west-2", "kind": "Queue", "namespace": "default", "name": "ackqueue-0", "generation": 2, "error": "Queue.sqs.services.k8s.aws \"ackqueue-0\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"foregroundDeletion\"}"}
2023-04-03T16:05:19.985+0200	ERROR	Reconciler error	{"controller": "queue", "controllerGroup": "sqs.services.k8s.aws", "controllerKind": "Queue", "Queue": {"name":"ackqueue-0","namespace":"default"}, "namespace": "default", "name": "ackqueue-0", "reconcileID": "ed30c91e-4bd2-456d-84ab-f0843b31902f", "error": "Queue.sqs.services.k8s.aws \"ackqueue-0\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"foregroundDeletion\"}"}

Without digging any deeper, this clearly an outcome of the controller trying to patch a new finalizer when it should not...

To reproduce this issue:

Spin an sqs controller with development logging level
Create 100 queues using https://github.com/aws-controllers-k8s/community/issues/1707#issuecomment-1472299763
Delete the queues using kubectl delete queue --cascade=foreground --all
If it doesn't work, try with more queues

Apr 03 '23 17:04 a-hilaly

This is a valid issue but given current resourcing, we're going to move this to important-longterm and in the short term we will update the current docs to indicate that foreground cascading is not currently supported.

Apr 12 '23 17:04 jljaco

@jljaco This may help some other folks using ArgoCD: we found that using the background finalizer resolves this problem. The documentation is a bit lacking, but after some experimentation, we found that specifying a finalizer like so does the trick:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io/background
...

Jun 21 '23 00:06 jessebye

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

Dec 18 '23 06:12 ack-bot

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

Jun 25 '24 01:06 ack-bot

Stale issues rot after 60d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 60d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle rotten

Aug 24 '24 02:08 ack-bot

/remove-lifecycle rotten

Aug 24 '24 14:08 jessebye

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

Feb 20 '25 19:02 ack-bot

Stale issues rot after 60d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 60d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle rotten

Apr 21 '25 20:04 ack-bot

Rotten issues close after 60d of inactivity. Reopen the issue with /reopen. Provide feedback via https://github.com/aws-controllers-k8s/community. /close

Jun 20 '25 20:06 ack-bot

@ack-bot: Closing this issue.

In response to this:

Rotten issues close after 60d of inactivity. Reopen the issue with /reopen. Provide feedback via https://github.com/aws-controllers-k8s/community. /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Jun 20 '25 20:06 ack-prow[bot]

/reopen

Jun 20 '25 22:06 knottnt

@knottnt: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Jun 20 '25 22:06 ack-prow[bot]

Rotten issues close after 60d of inactivity. Reopen the issue with /reopen. Provide feedback via https://github.com/aws-controllers-k8s/community. /close

Aug 20 '25 03:08 ack-bot

@ack-bot: Closing this issue.

In response to this:

Rotten issues close after 60d of inactivity. Reopen the issue with /reopen. Provide feedback via https://github.com/aws-controllers-k8s/community. /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Aug 20 '25 03:08 ack-prow[bot]