[RDS Controller] can't remove deletion protection

[gecube]

1. instantiate DB with FluxCD from typical manifest with deletionProtection: true
2. wait till DB would be created
3. remove DB from FluxCD control repo
4. the DB won't be removed from the cluster because deletionProtection: true so we have discrepancy between gitrepo and the actual state
5. The DB WOULD NEVER removed even if I would try to manually patch the deletionProtection: false

https://github.com/user-attachments/assets/d2fd1c22-0654-4352-95f5-f08fa64c467d

[github-actions[bot]]

Hello @gecube 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[gecube]

I was able to remove database when following the next steps:

• remove removal protection from the Amazon UI
• then instantly patch the object in k8s to deletionProtection: false
• and then the DB was removed

[gecube]

also I checked that the removal of deletion protection on RDS does not help, it should be followed by changing the flag to deletionProtection: false

[ack-bot]

Issues go stale after 180d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 60d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://github.com/aws-controllers-k8s/community. /lifecycle stale

[gecube]

/remove-lifecycle stale

[rushmash91]

Hey @gecube,

Thanks for reporting this issue. You've identified a deadlock in the deletion flow when deletionProtection: true is set on a DBInstance resource.

Currently whats happening is:

1. Deletion is initiated - When the DBInstance is removed from FluxCD (or deleted via kubectl delete), Kubernetes immediately sets metadata.deletionTimestamp on the resource
2. Controller attempts deletion - The ACK controller tries to delete the AWS RDS instance, but AWS responds with: InvalidParameterCombination: Cannot delete protected DB Instance, please disable deletion protection and try again
3. Controller enters terminal state - The controller sets ACK.Terminal: true and stops reconciling because it cannot proceed
4. Updates are blocked - Once deletionTimestamp is set, Kubernetes blocks updates to the spec (including changing deletionProtection: false) because resources in deletion are considered immutable

https://github.com/aws-controllers-k8s/runtime/blob/main/pkg/runtime/reconciler.go#L469-L492

5. Deadlock - The resource cannot be updated to remove deletion protection, and it cannot be deleted while deletion protection remains enabled

Workaround: What you currently did is a valid workaround, to manually change them via the console or cli.