Tutorial cannot be cleaned up - Timeout waiting for stack operation to complete

[esetnik]

Summary

Tutorial at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_tutorial_fargate.html leaves a CloudFormation stack which cannot be deleted. It might be related to https://forums.aws.amazon.com/thread.jspa?threadID=134440 which mentions

You need to add a dependency ("DependsOn": "AttachGateway") for a resource that depends on AWS::EC2::VPCGatewayAttachment, such as AWS::EC2::EIP. I had the same problem, which surfaced when parallel stack processing was introduced recently and this has fixed it permanently.

The specific failing resources are:

18:37:21 UTC-0400 | DELETE_FAILED | AWS::EC2::VPCGatewayAttachment | AttachGateway | Network vpc-0a97789633337db81 has some mapped public address(es). Please unmap those public address(es) before detaching the gateway. (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation; Request ID: 19066485-e256-47ec-8230-4e1a4c138d78)
-- | -- | -- | -- | --
  | Physical ID:amazo-Attac-1X09L1N4FHR29
  | 18:35:12 UTC-0400 | DELETE_FAILED | AWS::EC2::Subnet | PubSubnetAz1 | The subnet 'subnet-0006302161fe157e0' has dependencies and cannot be deleted. (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation; Request ID: 92b1e9c8-b146-4839-945f-7a3913e4f042)
  | Physical ID:subnet-0006302161fe157e0

Description

Command:

First follow all tutorial instructions...
❯ ecs-cli down --force --cluster-config tutorial
...
FATA[0760] Error executing 'down': Timeout waiting for stack operation to complete

• Which version of the CLI you are using (Run: ecs-cli --version)

❯ ecs-cli --version
ecs-cli version 1.8.0 (33b5630)

• Which version of Go are you using? (Run: go version)

❯ go version
go version go1.11 darwin/amd64

• What platform are you using to run ECS CLI commands?

macOS 10.13.6

Config files

• docker-compose.yml

version: '3'
services:
  wordpress:
    image: wordpress
    ports:
      - "80:80"
    logging:
      driver: awslogs
      options: 
        awslogs-group: tutorial
        awslogs-region: us-east-1
        awslogs-stream-prefix: wordpress

• ecs-params.yml

version: 1
task_definition:
  task_execution_role: ecsTaskExecutionRole
  ecs_network_mode: awsvpc
  task_size:
    mem_limit: 0.5GB
    cpu_limit: 256
run_params:
  network_configuration:
    awsvpc_configuration:
      subnets:
        - "subnet ID 1"
        - "subnet ID 2"
      security_groups:
        - "security group ID"
      assign_public_ip: ENABLED

• ~/.ecs/config

version: v1
default: tutorial
clusters:
  tutorial:
    cluster: tutorial
    region: us-east-1
    default_launch_type: FARGATE

Expected Behavior

The CloudFormation stack and all tutorial resources should be cleaned up

Observed Behavior

INFO[0000] Waiting for your cluster resources to be deleted...
INFO[0001] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0062] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0122] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0183] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0244] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0304] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0365] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0426] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0486] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0547] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0608] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0669] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
INFO[0730] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
FATA[0760] Error executing 'down': Timeout waiting for stack operation to complete

[GBmonmon]

I have this problem as will.

[PettitWesley]

@GBmonmon @esetnik This issue is quite peculiar. I ran through the tutorial steps just now and I encountered the same problem. However, when I wrote the tutorial about a year ago, I never recall this problem occurring.

Strangely, I was able to resolve it very easily. So when you do ecs-cli down it just deletes the CloudFormation stack, so CloudFormation deletes your resources for you. When I that just now, all of the resources were deleted, except for the VPC, which CloudFormation complained had dependencies and could not be deleted. Did you two see the same thing? I was able to clean up the stack by doing this:

1. Going to VPC Console and deleting the VPC. For some reason this worked.
2. Deleting the CloudFormation stack again, which succeeded now that the VPC had been deleted.

I am not really sure why CloudFormation couldn't delete the VPC when I was able to delete it using the VPC Console. I am also not sure why I've never encountered this problem outside of the tutorial just now- there's nothing special about the tutorial, it just creates a cluster and resources configured for Fargate. The CLI creates a very similar set of resources for EC2 clusters as well. Have either of you ever encountered this issue outside of the tutorial or us-east-1?

[DevopsDood]

Same issue

[scottstensland]

to delete vps use browser since cli fails to handle this see add --all-dependencies option to ec2 delete-vpc #1721 https://github.com/aws/aws-cli/issues/1721