aws-genai-llm-chatbot icon indicating copy to clipboard operation
aws-genai-llm-chatbot copied to clipboard

'npx cdk destroy' failed because ENI was in use

Open alexeyshishkin01 opened this issue 1 year ago • 8 comments

Hello, everyone !

My command 'npx cdk destroy' failed because ENI was in use during the deletion.

So I had to manually delete ENI, then subnet, then VPC, then Cloud Formation stack.

Could you pls. have a look at the code to fix the issue ?

best regards Alexey Shishkin

alexeyshishkin01 avatar Feb 01 '24 20:02 alexeyshishkin01

Does the force option help?

npx cdk destroy --all --force

scottschreckengaust avatar Feb 01 '24 22:02 scottschreckengaust

I don't know. To test this option I have to launch the solution again and try to destroy it.

Is it possible to update the documentation https://aws-samples.github.io/aws-genai-llm-chatbot/guide/deploy.html#clean-up with this option ?

alexeyshishkin01 avatar Feb 01 '24 22:02 alexeyshishkin01

Just did a full deployment with each model and RAG. When I went to destroy, two resources were marked as deleted but in fact not actually delete.

  1. Sagemaker Endpoint (IDEFICS)
  2. VPC Endpoint (shared vpc)

No solution in cdk yet, just stating I'm seeing the same thing. I had to go and find through through resources in cloudformation stack, delete them manually, then go back to cloudformation to complete the deletion.

QuinnGT avatar Feb 12 '24 08:02 QuinnGT

Does the force option help?

npx cdk destroy --all --force

It looks like it does not help as well:

3:36:49 PM | DELETE_FAILED | AWS::EC2::Subnet | SharedVPCprivateSubnet1Subnet5A4C2616 Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest)

❌ genaiGenAIChatBotStack: destroy failed Error: The stack named genaiGenAIChatBotStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [SharedVPCprivateSubnet1Subnet5A4C2616]. ): Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest) at destroyStack (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:428:2204) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async CdkToolkit.destroy (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:431:204632) at async exec4 (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:486:54250)

The stack named genaiGenAIChatBotStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [SharedVPCprivateSubnet1Subnet5A4C2616]. ): Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest)

The subnet has an ENI which is not deleted. The ENI Description is 'SageMaker managed ENI for endpoint amazon-FalconLite instance i-03b16c72c5b18266d'

alexeyshishkin01 avatar Mar 22 '24 18:03 alexeyshishkin01

these commands can be helpful for the troubleshooting:

aws ec2 delete-network-interface --network-interface-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite*" --query 'NetworkInterfaces[*].NetworkInterfaceId' --output text)

aws ec2 delete-subnet --subnet-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite*" --query 'NetworkInterfaces[*].SubnetId' --output text)

alexeyshishkin01 avatar Apr 04 '24 13:04 alexeyshishkin01

as a workaround one can run the following in the script:

npx cdk destroy --all --force aws ec2 delete-network-interface --network-interface-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite*" --query 'NetworkInterfaces[].NetworkInterfaceId' --output text) aws ec2 delete-subnet --subnet-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite" --query 'NetworkInterfaces[*].SubnetId' --output text) npx cdk destroy --all --force

alexeyshishkin01 avatar Apr 11 '24 18:04 alexeyshishkin01

it looks like it's an old issue in AWS - https://stackoverflow.com/questions/52416454/sagemaker-model-cloudformation-stack-deletion

alexeyshishkin01 avatar Apr 25 '24 13:04 alexeyshishkin01

This issue is stale because it has been open for 60 days with no activity.

github-actions[bot] avatar Jul 16 '24 01:07 github-actions[bot]

This issue was closed because it has been inactive for 30 days since being marked as stale.

github-actions[bot] avatar Aug 16 '24 01:08 github-actions[bot]