Make hooks a part of the dependency graph.

[ejholmes]

At the moment, the lifecycle around hooks are pretty basic:

1. pre_build runs before stacker does anything to stacks
2. post_build runs after stack does everything to stacks

It would be nice if hooks were just a part of the dependency graph, like stacks are, so that you can reference outputs from hooks, and use them as inputs to stacks and vice/versa.

For example:

hooks:
  - name: "ecsCluster"
    class_path: stacker.hooks.ecs.create_clusters
    args:
      clusters:
        - "staging"
          
stacks:   
  - name: "app"
    class_path: blueprints.App
    variables:
      ECSCluster: ${hook_data ecsCluster::ClusterName}

Instead of stacker executing the hooks in a separate step, stacker would just execute the hooks as it walks the dependency graph (app depends on ecsCluster).

I'm not even sure if "hooks" would be the right name for this if it was supported, since it'd be more like a pseudo/virtual stack.

[phobologic]

So I've been thinking about this - a couple of thoughts:

The easiest way to get this in is to likely make it just part of the stacks config directly - basically your idea of a virtual stack. It could just be a replacement for the Blueprint type - a Hook class maybe?

We can continue to use variables just like we do now, though we'd have to blow up on CFNTypes, which shouldn't be too hard.

The other interesting part of this is that then those classes could have a create and a destroy mode, that you could optionally enable/disable. This makes a ton of sense, honestly - one of the hooks I really would love to have available that isn't really easy right now, would be an s3 sync type hook, which can wait on a bucket to be created, then upload contents from a directory to it. It'd be the last step in a fully managed "Here's how to create and maintain an S3 website in a single stacker config" concept. In that case, if you were going to destroy all the stacks, you likely want to destroy everything in the bucket, since S3 won't let you delete a bucket until it's empty.

The only weird part is the name of the top level config stacks - but maybe that's not such a huge deal.

Anyway, any thoughts?

[danielkza]

@phobologic I can think of quite a few different ways to move forward, some more pragmatic than others. I think all of them are already would already major improvement to the status quo.

a) Add pre_build, post_build, pre_destroy and post_destroy optional methods to blueprints and encourage people to move hook code there when applicable. That would be the best fit with how I use most of the hooks, which is to provide capabilities that CFN omits or do local work to generate data required for a template. Handling Lambda functions, for example, would be much easier like this.

b) Add a hooks option at the top-level. Mostly re-use the current Hook definition, but resolve outputs in the arguments, and add a requires key. Internally merge their definitions in the DAG so everything runs neatly. Possibly deprecate pre_build and post_build.

c) Leave hooks as is, and provide a framework for easily writing and deploying custom CloudFormation resources instead. Would be a large project and completely different from the current usage patterns, but would be the "ideal" solution for many cases, and provide major benefits that hooks can never achieve, such as non-incorruptible deployments, rollbacks, independence from Stacker, etc.

I'm personally very interested in c, but I recognize it might not be feasible in the short term. Given how Terraform has been (justifiably) getting increasingly popular, I feel Stacker can stay relevant by playing to CloudFormation's strengths that TF can't match (mostly regarding deployment reliability), instead of moving closer to TF (by doing more work locally)