elasticache-controller - issue deleting `CacheParameterGroup`

[ferozemohideen]

Describe the bug I was testing the elasticache-controller, and I have found that when I run kubectl delete on the following template (attached), all resources are deleted except for the CacheParameterGroup, whose delete fails because of the following condition:

Status:
  Ack Resource Metadata:
    Arn:               arn:aws:elasticache:us-west-1:*********:parametergroup:test
    Owner Account ID:  *********
    Region:            us-west-1
  Conditions:
    Last Transition Time:  2024-01-29T17:19:01Z
    Message:               Resource synced successfully
    Reason:
    Status:                True
    Type:                  ACK.ResourceSynced
    Message:               InvalidCacheParameterGroupState: One or more cache clusters are still members of this parameter group test, so the group cannot be deleted.
                           status code: 400, request id: b189ad54-88bc-4b4a-ae4d-1cb35c4f3e90
    Status:                True
    Type:                  ACK.Terminal

discussion on Slack: https://kubernetes.slack.com/archives/C0402D8JJS1/p1706552666549969

Steps to reproduce Apply the attached template, then delete Expected outcome Expected the CacheParameterGroup CRD to be deleted Environment

• Kubernetes version
• Using EKS (yes/no), if so version? yes, v1.27.0
• AWS service targeted (S3, RDS, etc.) ElastiCache

template: delete.txt

[davejfranco]

I just had the same issue. I'm using it inside a helm chart, so when I try to delete the chart the ParameterGroup gets stuck, I think is because by the moment it tries to delete it, the ReplicationGroup is still active. After the ReplicationGroup is gone I tried to delete it manually with no success.

[a-hilaly]

@davejfranco We're working on fixing some flaky tests/security patches first. We'll dive into this one ASAP.