CloudWatchLogsLogGroup - stuck waiting

[mbergal-idx]

/aws/eks/idx-eks-pulumi-test-eksCluster-6586a80/cluster - waiting Removal requested: 1 waiting, 0 failed, 74 skipped, 95 finished 
us-east-2 - CloudWatchLogsLogGroup - /aws/eks/idx-eks-pulumi-test-eksCluster-6586a80/cluster - waiting Removal requested: 1 waiting, 0 failed, 74 skipped, 95 finished

Not sure what to add, but I able to delete this log group manually

[mbergal-idx]

Maybe because when request to delete it is made, EKS cluster is still writing to it?

[rbbrasil]

@mbergal-idx which release are you running?

I've just got this problem with the v2.14.0. With v2.14.0 the removal went fine.

ap-southeast-2 - CloudWatchLogsLogGroup - /aws/vpc/mgmt-VPC/flow-logs - waiting

Removal requested: 1 waiting, 0 failed, 68 skipped, 0 finished

ap-southeast-2 - CloudWatchLogsLogGroup - /aws/vpc/mgmt-VPC/flow-logs - removed

Removal requested: 0 waiting, 0 failed, 68 skipped, 1 finished`

[rajivchirania]

@mbergal-idx Your above comment contains the same version.. I am also using the version v2.14.0 and having the same problem with eks cluster deletion and cloudwatch log group deletion..

[rbbrasil]

Oops! That was a typo. Sorry.

The Docker image version that worked for me was 2.12.0.

[mbergal-idx]

I am using 2.14.0, deleting single log group works fine, but if log group is associated with the cluster it does not get deleted for some reason.

[svenwltr]

Hello.

Sorry for the late response. Can you give us a hint to to reproduce this error?

[mbergal-idx]

This happens if this is EKS cluster's log group. I think it gets deleted but cluster recreates it since it takes more time for it to get deleted. If this is not enough I might be able to create a simple pulumi script to make a repro.

[rickepnet]

I have the same issue EKS cluster create a cloudwatch log group if I delete the log group by hand it deletes just fine. v2.13.0 I will try to upgrade see if it helps.

[jbarnes]

👍🏻 on this issue. Restarting the tool against the account in question, the log group is successfully deleted no problems.

This is a bit of an issue when using automation as the tool will simply recycle the deletion process and continue to fail. Not sure if I can set a retry count and exit with error or not but that would be preferable.

v2.14.0 is the version I am using, also in ap-southeast-2 region.

[artem-nefedov]

We're also seeing this issue, and it's also caused by EKS log groups.

Reproduction steps should be:

1. Create any cluster with eksctl tool and make sure to enable control plane logging to cloudwatch
2. Run aws-nuke on the account

Probably the important part is that cluster still exists when aws-nuke is run, and is also deleted in the process. Log groups from previously deleted clusters do not cause this issue.

[svenwltr]

I think it gets deleted but cluster recreates it since it takes more time for it to get deleted.

I follow this idea. Unfortunately I do not see a obvious solution to this.

[ivan-sukhomlyn]

I have the same situation with the 2.15 version. CW Log Group is in a stuck state. Logs:

13:35:15  us-west-2 - CloudWatchLogsLogGroup - /aws/eks/{ cluster_name }/cluster - waiting
13:35:15  
13:35:15  Removal requested: 1 waiting, 0 failed, 147 skipped, 64 finished
13:35:15  
13:35:20  us-west-2 - CloudWatchLogsLogGroup - /aws/eks/{ cluster_name }/cluster - waiting
13:35:20  
13:35:20  Removal requested: 1 waiting, 0 failed, 147 skipped, 64 finished
13:35:20  
13:35:25  us-west-2 - CloudWatchLogsLogGroup - /aws/eks/{ cluster_name }/cluster - waiting
13:35:25  
13:35:25  Removal requested: 1 waiting, 0 failed, 147 skipped, 64 finished

[wushingmushine]

Still seeing aws-nuke (v 2.17) indefinitely hanging when deleting cloudwatch log groups. If I cancel the aws-nuke run and re-run, the log group deletes immediately without issue.

This is unrelated to EKS for me.

Recreate using boto3 to create log group:

mySession =  boto3.Session(
    aws_access_key_id=accessKeyId, aws_secret_access_key=secretAccessKey
)
logsClient = mySession.client('logs', region_name='eu-west-1')
log_group_response = logsClient.create_log_group(
    logGroupName='all-rejected-traffic'
)

in eu-west-1 in a completely wiped account. The log group is then targeted by a flow log on the default VPC. Then running aws-nuke gives

Removal requested: 1 waiting, 0 failed, 296 skipped, 21 finished

eu-west-1 - CloudWatchLogsLogGroup - all-rejected-traffic - waiting

Removal requested: 1 waiting, 0 failed, 296 skipped, 21 finished

eu-west-1 - CloudWatchLogsLogGroup - all-rejected-traffic - waiting

Removal requested: 1 waiting, 0 failed, 296 skipped, 21 finished

eu-west-1 - CloudWatchLogsLogGroup - all-rejected-traffic - waiting

It seems aws-nuke is getting stuck in some indefinite cycle?

[leiarenee]

Infinitely waiting aws-nuke version 2.22.1

eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - failed
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/otomi/cluster - [CreatedTime: "168334--183", LastEvent: "2023-05-06T06:20:34+02:00", logGroupName: "/aws/eks/otomi/cluster", tag:created_by: "terragrunt", tag:k8s: "custom", tag:workspace: "testing"] - waiting
eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting

Removal requested: 2 waiting, 2 failed, 93 skipped, 181 finished

after re-run it terminates gracefully.

[chronicc]

Still failing with v2.22.1.15.ge45750a

eu-central-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - failed
eu-central-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting

Removal requested: 1 waiting, 1 failed, 557 skipped, 1387 finished

[martivo]

Still same issue, using: quay.io/rebuy/aws-nuke:v2.23.0 I have two eks clusters, each have one CloudWatchLogsLogGroup. One of them always gets deleted, the other stays in infinite loop.

[nileshgadgi]

I am facing the same issue also, my github workflow took 6 hour to delete the resources, It automatically got cancelled due to default timeout, it was stuck in the CloudWatchLogGroup Deletion.

Github action gives 3000 minutes free for free organization account and I'm getting the same error every time and I have to compromise with github time limits.

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

eu-west-1 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - waiting
eu-west-1 - CloudWatchEventsRule - Rule: AutoScalingManagedRule - waiting
eu-west-1 - CloudWatchLogsLogGroup - /aws/eks/eks-velero-cluster/cluster - [CreatedTime: "1695364752907", LastEvent: "2023-09-22T22:38:20Z", logGroupName: "/aws/eks/eks-velero-cluster/cluster", tag:Environment: "velero", tag:Managedby: "[email protected]", tag:Name: "eks-velero-cluster", tag:Repository: "https://github.com/clouddrove/terraform-aws-eks"] - waiting

Removal requested: 3 waiting, 0 failed, 930 skipped, 105 finished

[lucazz]

Tagging along with the same issue here. Tearing down EKS clusters and their respective dependencies fails because it can't delete the Cloudwatch log group.