aws-cdk icon indicating copy to clipboard operation
aws-cdk copied to clipboard

cli: cdk deploy returns ok (exit code 0) if DELETE_FAILED

Open jimduden6804 opened this issue 3 years ago • 1 comments
trafficstars

Describe the bug

I am using cdk deploy resources in an github actions pipeline. Creating I "new" resource may force us to allow the pipeline role to build the resource (e.g. cloudwatch:PutMetricAlarm) Removing this resource from the stack then needs the deletion action (e.g. cloudwatch:DeleteAlarm) Sometime we forget to add the deletion action when adding the build action.

If we later remove the construct from the stack it cannot be deleted and we do not notice because cdk deploy does not throw an error. The logs show cloudformation not able to delete the resource "DELETE_FAILED", but since the exit code of "cdk deploy" is 0 the pipeline continues.

Expected Behavior

"cdk deploy" returns exit code != 0 if DELETE_FAILED

Current Behavior

"cdk deploy" returns exit code = 0 if DELETE_FAILED

Reproduction Steps

How to reproduce:

  • I deploy a stack containing a cloudwatch alarm
  • I remove the alarm from the stack
  • I deploy the change (cdk deploy)
  • Cloudformation tries to delete the alarm but gets DELETE_FAILED (in this case because my deployment role is only allowed to create but not delete the alarm)
  • cdk deploy returns with exit code 0 and pipeline continues
  • cloudformation stack shows status "Update successful. One or more resources could not be deleted."

Possible Solution

"cdk deploy" returns exit code = 0 if DELETE_FAILED

Additional Information/Context

No response

CDK CLI Version

2.39.0

Framework Version

No response

Node.js Version

v18.0.0

OS

Ubuntu, iOS

Language

Typescript

Language Version

No response

Other information

No response

jimduden6804 avatar Sep 22 '22 14:09 jimduden6804

Might be a cloudformation issue which cannot be solved by cdk though.

relates to this issue: https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/744

jimduden6804 avatar Sep 22 '22 14:09 jimduden6804

DELETE_FAILED is a resource status. What is the stack status?

rix0rrr avatar Oct 10 '22 15:10 rix0rrr

Status: UPDATE_COMPLETE Description: Update successful. One or more resources could not be deleted.

jimduden6804 avatar Oct 11 '22 07:10 jimduden6804

Huh. Never knew that was a possibility.

But in any case, we mirror the CloudFormation Stack status. So if CFN says everything is hunky-dory, we don't argue.

Could be a feature request. What are you trying to achieve, though? Why is it important to you that the exit code indicates failure?

rix0rrr avatar Nov 08 '22 13:11 rix0rrr

We have a CI pipeline. I would like to achieve that there is no rogue infrastructure left which we have to pay and don't know about. In this case it was just a coincidence that we saw the message. We have way to many aws resource to check every single on belonging to a stack. If someone has an idea to monitor this issue, it would also help

jimduden6804 avatar Nov 08 '22 14:11 jimduden6804