aws cloudformation deploy does not clean up failed change set

[andreaswittig]

I'm using aws cloudformation deploy in a CI/CD pipeline with the option --no-fail-on-empty-changeset. This allows the pipeline to deploy all stacks, no matter if there was a change to the template or parameters.

However, aws cloudformation deploy with the option --no-fail-on-empty-changeset does create a change set which goes to state FAILED with reason The submitted information didn't contain changes. Submit different information to create a change set.. The problem is, that the failed change set does not get deleted. Therefore, hundreds of change sets pile up. Resulting in an LimitExceededException (ChangeSet limit exceeded for stack ...) after a while.

In my opinion, aws cloudformation deploy with option --no-fail-on-empty-changeset should delete change sets, that failed because of an empty change set. Or there should at least be an option to do so.

[joguSD]

I've reproduced this issue and can see how this would be less than ideal. My only concern with automatically deleting the failed change set is that it's technically a breaking change and I don't know if there are potentially customers relying on the failed change set being present.

@sanathkr Any thoughts on how to handle this?

[andreaswittig]

@joguSD Thanks for looking into this. I agree, that others might rely on the current behaviour. Although, that is not very likely. The change set does not contain any valuable information for debugging, in this scenario.

[atkinsonm]

To support users who rely on the current behavior while at the same time solving the problem for others, a new command line argument could be added (say, --cleanup-on-failure). The default behavior could remain the same as it does now. The new argument would only be relevant in cases where --no-fail-on-empty-changeset is also specified.

[mcallaghan-bsm]

There is a post explaining this in more detail fwiw: https://cloudonaut.io/aws-cli-cloudformation-deploy-limit-exceeded/

[mgorski-mg]

We had a problem with LimitExceededException so I wrote Lambda to clean it up and published it in SAR. Feel free to try! https://github.com/mgorski-mg/aws-failed-change-sets-destroyer

Of course, I'm waiting for a proper fix too! :)

[gagipro]

I encounter this issue now that we intensively use CDK with Nested Stack. everytime there's no change in a stack it creates a failed changeset in the Nested that you cannot even delete.

Only way to remove them is to create a change that updates the Nested Stack...

[Balou9]

Hi everybody.

I created a github action that might be helpful to solve this problem. https://github.com/Balou9/cfn-check-failed-status I was thinking about what you guys already mentioned: Is it wanted to have an automatic deletion of the failed change set?

For a real-world application that others rely on, maybe not. But for anything else (e.g. in the early stages of mvp programming), it might be annoying to always open the console or use the cli to delete the failed change set. At least you should have the possibility to do this automatically .

The idea is to use the action prior to the deployment step in a github actions pipeline. It checks for the stack status "FAILED" instead of failed change sets.

I would love to hear your opinion on this, especially on what needs to be improved (e.g. deleting nested stacks, check for failed change sets instead or as well as checking for failed stack status etc.)

One question that is still open for me: should this be handled by an action or an additional command line argument the aws cli?

[gagipro]

change

for the moment there is no possibility, that I'm aware of, to delete failed nested stack changeset.

so there should be this possibilty through sdk and aws cli.

[razvanphp]

This is really frustrating, please address this.

Here is a script to cleanup in bash with aws-cli: https://cloudonaut.io/aws-cli-cloudformation-deploy-limit-exceeded/

[Abhinab-AY]

Do we have any update or progress on this. This being such an important feature, it should have had much higher prio.

[mdbouchard]

We recently ran into this issue at my company. I had no idea --no-fail-on-empty-changeset caused empty FAILED change sets if our CI/CD pipeline attempted to deploy a stack which hasn't changed in a while. One of our stacks had 500 of these which needed to be cleaned up.

[gagipro]

We recently ran into this issue at my company. I had no idea --no-fail-on-empty-changeset caused empty FAILED change sets if our CI/CD pipeline attempted to deploy a stack which hasn't changed in a while. One of our stacks had 500 of these which needed to be cleaned up.

It happens when there's no change i think ?

[mdbouchard]

It happens when there's no change i think ?

Correct! We tend to have a prerequisite job for setting up a once-in-a-while change as part of our pipeline. When that almost never changes, the no-op (empty) changes pile up. Because of --no-fail-on-empty-changeset, we never see a failure in the job itself unless we reach 500 FAILED change sets that are empty.

I have observed that if we DO have a change set that's not empty, both aws cloudformation deploy and sam deploy proactively clean up past FAILED change sets, which is nice.

[gagipro]

It happens when there's no change i think ?

Correct! We tend to have a prerequisite job for setting up a once-in-a-while change as part of our pipeline. When that almost never changes, the no-op (empty) changes pile up. Because of --no-fail-on-empty-changeset, we never see a failure in the job itself unless we reach 500 FAILED change sets that are empty.

I have observed that if we DO have a change set that's not empty, both aws cloudformation deploy and sam deploy proactively clean up past FAILED change sets, which is nice.

I introduced a dummy resource (iam empty role if i remember) that changes everytime with a timestamp. That’s dirty but that’s how I decided to workaround this bug which have no fix since so long time... We have hundred’s of stacks and nested stacks. The same for the bug that prevents to see content of changeset in nested stacks, need to patch the cdk lib...and without nested stacks, you get really fast to 2000 stacks limits. Don’t even know if it will work if i update the cdk lib... Lots of bugs that makes production with cdk eeally difficult to maintain.

[colbydh]

I ended up writing a bash script which loops through and deletes failed changesets that pile up. If too many changesets stack up, cf stacks won't be able to deploy.

[gagipro]

I ended up writing a bash script which loops through and deletes failed changesets that pile up. If too many changesets stack up, cf stacks won't be able to deploy.

Should have been engineered from beginning or at least adressed since ...