Support more flexible interactive builds

[acmcelwee]

I started this discussion in the empire/#stacker slack channel yesterday, and it sounds like multiple teams have thought about various ways to allow for interactive builds, w/ stacker running somewhere other than on a privileged-IAM user's machine.

Why are people stated for still running interactive builds from their machine for critical stacks?

• Need fine-grained visibility into the CFN resources being changed as a final sanity check
• Security - the current setup allows for mandated MFA to run

Various ideas were discussed to achieve parts of the requirements that I have in mind:

• Stacker communicates changesets to a lambda function, w/ the expectation that changesets get executed out of band by an IAM user that has the necessary access level. Stacker would probably poll for a change to the changeset (execute or delete) and would then resume processing the temporarily paused DAG plan.
• More generic "approval plugins" that, instead of doing IO on STDOUT/STDIN, it would delegate to a python class, specified in the stacker config.

[acmcelwee]

The first idea of changesets->lambda and polling for changes to a changeset seems like it provides both of the stated needs of fine-grained visibility and security.

It looks like all of the necessary hooks are in place to override the approval prompt and waiting behavior, so it would probably just be a matter of defining the expected payload to send to the lambda. I'd imagine some sort of context identifier for what triggered the stacker run and the changeset arn. That arn should give all of the context that someone would need (account, region, stack name).

[acmcelwee]

On second thought, https://github.com/remind101/stacker/issues/569#issuecomment-380605715, we need prompt handling for other things that won't have a changeset, like prompting before delete-before-recreate and possibly prompting before creating new stacks.

[ejholmes]

I think I would be in favor of the more generic "plugin" system; basically just making stacker.ui overridable. I think it'd be hard for stacker to support every CD workflow that people want, so just giving them the flexibility to do it however they see fit might be best.

It may also be worth pointing out that another way to do this is just use a little unix/bash magic and redirect STDOUT/STDIN to another process.

stacker build stacker.yaml > >(stdout-proc) < <(stdin-proc)

[brettswift]

I'm not sure where we'll end up with our CD flow but the initial thought of a simple CodePipeline process, where stacker could create the changeset, and then execute it.. under two separate commands.. would be helpful.

That way we could easily view what resources are being changed.

This is a good sanity check for those that don't have 100% reliable continuous testing.. With a DEV release cadence that might be more frequent than PROD .. your delta might accumulate one or two DEV releases.

So seeing what resources are going to change - just as a sanity check would be nice.

I like the pluggable option, where the different stages of stacker build could be customized. Right now the only option I see is to run stacker build` in code pipeline and force it through.

[ejholmes]

@brettswift something along the lines of this could be cool. I think, at the very least, support for stacker to continue execution if the changeset is manually approved through other means (e.g. someone goes into CloudFormation console and applies the changeset) would be good.

One problem with a multi phase solution like you suggest above is that you may not completely capture all the changes that would actually happen. For example, if you have a stack B that depends on an output from A, and then change that output in A, you could only really capture the changeset from A, without knowing that B is going to change. In some CD workflows, this may be good enough.

[brettswift]

I'd like to add a few more thoughts - possibly from a different perspective.

Executing things out of band is scary to me, because of the A/B stack dependencies and execution order that you mentioned above @ejholmes. Nested stacks force this on you - for good reason..but there are drawbacks of course (that's why we're here!). With the flexibility of stacker we would want to maintain robustness and consistency of stack deploys. Today it's tough to get into a "half baked" situation with stacker build. I tend to want to say "if stacker creates the changeset, stacker must execute the changeset", and a user shouldn't go to cloudformation to do it.

The moment we create a changeset and leave it there, someone could trip us up. But stacker could control that more consistently. I'm suggesting it should.

The simple MVP I'd probably support the most are three new commands: (better names maybe!)

stacker create-changesets
stacker describe-changesets (accumulates all changesets in all stacks)
stacker execute-changesets

If this was in a pipeline you could set up policies to only allow your pipeline to execute the changesets. In effect solving the "out of band" risk. (A more complicated option would be to involve cloudformation wait handles, so the changeset would already be executed and it would just be waiting for stacker to let it continue.. but that's going to be very involved).

If your pipeline can execute stacker create-changeset it should be able to stacker execute-changeset. I'm kind of missing the need for the lambda. Your pipeline (I would think) could control notifications and just let the user go to cloudformation to view the changeset.. or a pipeline dashboard could make calls to stacker describe-changeset that would be handy on the CLI as well. (I'm really wanting cli / pipeline parity).

My motivations:

• simplicity
• no state
• no polling etc

I feel like I'm missing the need for the lambda though. Feel free to explain it here or ping me on the slack channel. (might just be bswift there? )

My primary interest in stacker is for pipelines. And I'm not sure I could "ease" an ops group into this without a separate create and execute step. So I'm quite interested in helping out here.

Again, just my perspective. Others surely have different use cases. Possibly the simplest solution wouldn't get everyone what they want, but could give everyone a little bit, and be enough to build on?