Delineation of env/service CF template executions

[AndreyMarchuk]

Problem statement

Resources are defined in separate mu.yml each:

• environment X
  • uses an extension to provision new IAM role (common-iam CF template) which can be used by service B
• service A
  • uses an extension to provision DynamoDB tables
  • extension updates common-iam CF template to allow CF role to create/modify/delete DynamoDB tables
• service B
  • uses an extension to provision SQS queue
  • extension updates common-iam CF template to allow CF role to create/modify/delete SQS queues

Case 1

• service A and service B are being built by CI/CD at the same time
• both of them need to update common-iam CF template/stack before provisioning the resources
• potentially this creates a bottleneck in CI/CD performance due to each stack waiting for common-iam update (think 20 services built at the same time)
• additionally if service A needs a lot of time to execute other CF templates after common-iam update (i.e. provision DynamoDB), service B might be updating common-iam at the same time resulting in revocation of DynamoDB provisioning permissions from common-iam and deploy failure

Case 2

• service B starts a build right after service A finished building
• since service A does override common-iam template, all the overrides to the same template on environment level are lost (IAM role provisioned on env level will be gone)
• thus service B build pipeline discovers missing IAM role and fails

Potential solution

• use global extensions/overrides on the environment level (i.e. common-iam)
• do not execute environment level CF templates when doing svc deploy, pipeline up etc

Please let me know your thought on these issues and ideas about potential resolutions.