Investigate support for deleting stacks that have been removed from configuration

[danielkza]

Adding a stack to a config file, deploying it, then removing it will currently leave it "orphaned" if you forget to run stacker destroy beforehand. It can, of course, be manually deleted using the AWS Console, but it's easy to miss it when you have hundreds of stacks.

It would be nice to have some way to enforce that a config contains all the stacks that represent one service. It would pair nicely with adding Nested Stacks support, such that a config could be represented by the parent stack, with all contained stacks as its children. Updates would go through the parent stack and remove any stacks that are no longer present.

[brettswift]

I'm just getting familiar with stacker in the last week. I could see something like a stacker cleanup command or stacker build --no-cleanup and make cleanup the default behaviour. (I kind of thought it might do that already).

One of the main reasons I'm experimenting with stacker is because I don't like nested stacks. Just my 2 cents, but it seems like stacker was built as an alternative to exporting / importing resources, and to nested stacks.

[danielkza]

@brettswift stacker cleanup (and/or a --cleanup flag for stacker build) seems like a good solution to deal with orphans without breaking existing assumptions.

I share your dislike of nested stacks - I tried them briefly and found them not to be nearly as flexible as stacker. I don't want the existing lookup/dependency mechanisms to be replaced, but just to use the nested stacks to "persist" which stacks belong to which "compartments", such that config file changes can never lose information about it.

I can think of other approaches, such as using tags to represent a flattened version of the dependency DAG, and cleaning up based on that. Being able to adapt existing stacks to support cleanup would be a very distinct advantage over nested stacks (I couldn't find any reference to importing existing stacks as children of another; is it possible at all?).

I'm open to any other suggestions so that I can experiment with them, since I plan on implementing this in the near future.

[ejholmes]

I was thinking a bit about this recently. Essentially, what this boils down to is storing some kind of state somewhere. In addition to making deletions like this easier, it could also be used to cache Outputs for stacks, which can drop stacker build times pretty significantly (all the perf problems in stacker are because of DescribeStacks performance).

I think if we implemented this, cleanup should just be a part of stacker build; it's just another mutation on the graph. We can model it like how CloudFormation does cleanups, where it happens in a "cleanup" phase at the end of the run (and prompts you if using --interactive).

[phobologic]

I'm open to providing this functionality - I think @ejholmes is likely right, we'd need to store state between runs. I'm always hesitant to automatically, or by default, delete any stack - that is just leaving someone open to making a mistake and destroying something, and that's a bad reputation for stacker to get, as an infrastructure tool.

If we made destroying deleted stacks part of stacker build, I think we should probably start with it automatically falling into interactive mode for those deletes. I feel like this happens so infrequently that being safe here for a while is likely the right call.

[russellballestrini]

I really like that stacker doesn't keep any state except for what CloudFormation stores. Personally I like that stacker doesn't delete stacks by default.

[danielkza]

@russellballestrini I would absolutely prefer to not have to introduce any additional state that requires complex management. So far Tags are the solution I'm leaning to, simply because they are easy to handle and don't require any large changes other than some API call parameters.

Not deleting by default is also perfectly fine by me, specially since changing it would be a major unexpected API breakage. But I also found myself with orphan stacks multiple times in the past two years, and I would bet most other heavy Stacker users did too (even if they did not notice it).