Tag tasks to support staged deployments

[rene84]

We have been searching for the right way to do staged deployments and have so far not come up with a satisfactory solution.

The biggest underlying problem, I believe, is that we are using org-formation to manage the organization as being the platform. Since teams running their workloads on it (which do have a clear DTAP) then from their perspective all of the platform should be considered production. However, as the resources we manage within org-formation grew, there are more complex things in there such as EKS clusters and reverse proxy lambdas which are critical and actually do have several instances across the platform which we from a platform team perspective consider separate environments (DTAP). Some approaches:

1. Set up a completely separate organization. This is the cleanest solution, but has some drawbacks such as that it can become costly (in multiple ways), doesn't have all the feature team workloads on it so is less representative

2. Extract those resources we want to deploy staged into a separate repository. Following the example, we could deploy the EKS clusters from a separate repository, but we'd miss the benefit of cross-account deployments and dependency resolution that a single super-repo with org-formation gives us.

3. Tag org-formation tasks as DTA and execute separate builds executing only tasks based on those tags. We use pull requests to master in our peer-reviewing process and already have a build with validate-tasks on the branch to check the correctness of the change. If we add another step to that build to execute-tasks for those tasks only tagged, then I believe we'd have a concise solution to this problem.

4. ...

I'm interested in a discussion about alternatives that could work better or have a different trade-off as well as challenges to be resolved for solution 3 to be viable.

Btw: so far we have been doing a few things to mitigate this:

• (ab)use the 'Skip' feature to merge to master to skip PRD execution, test, unskip and run again. This is very error-prone
• deploy from the local machines to lower environments. Sort of works, but is manual and requires some setup on the local machine and of course isn't visible to other team members

[eduardomourar]

I believe we could leverage #130 to help us achieve option number 2, but I still think we would need tag deployment. My suggestion is to have way to deploy based on tag by passing a regular expression so that you create a deployment pipeline "for PRD" and another one "NOT for PRD".

[rene84]

Right, because we already sort of by convention put the environment in the Task logical name whenever we decided they needed to be tagged that way. Where we don't have that separation the task logical name doesn't contain things like Dev, Prd or Dta at the beginning or end

[OlafConijn]

so if i understand correctly, something like this

FeatureDev:
  Type: include
  DependsOn: FeatureCore
  Path: ./feature-tasks.yml
  Parameters:
    resourcePrefix: !Sub "${resourcePrefix}-dev"
  Tags: [DEV]

FeatureProd:
  Type: include
  DependsOn: FeatureCore
  Path: ./FeatureCore-tasks.yml
  Parameters:
    resourcePrefix: !Sub "${resourcePrefix}-prd"
  Tags: [PRD]

then run org-formation perform-tasks using an option similar to --exclude-tasks-with-tag PRD.

Should this not be more like the env option in serverless? where you pass the environment and use the environment variable to something like:

Feature:
  Type: include
  DependsOn: FeatureCore
  Path: ./FeatureCore-tasks.yml
  Parameters:
    resourcePrefix: !Sub "${resourcePrefix}-${opt:env}"

for features that should not be released to production, something like this:

Feature:
  Type: include
  Skip: !Equals [!Ref opt:env, "prd"]
  DependsOn: FeatureCore
  Path: ./FeatureCore-tasks.yml
  Parameters:
    resourcePrefix: !Sub "${resourcePrefix}-${opt:env}"

would be interested to hear what you prefer and why!