aws-deployment-framework icon indicating copy to clipboard operation
aws-deployment-framework copied to clipboard

deployment account bootstrap step function CreateOrUpdateBaseStack timeout

Open AndreasAugustin opened this issue 3 years ago • 2 comments

Hi,

ADF version version 3.1.2

Describe the bug When bootstrapping the deployment account with 3 target regions the step function step CreateOrUpdateBaseStack has a timeout.

To Reproduce Create ADF framework with SAR

  • set 3 DeploymentAccountTargetRegions
  • either move the deployment account to the deployment OU or let SAR create the deployment account
  • The step function step CreateOrUpdateBaseStack receives a timeout

Expected behavior The provisioning of the deployment account finishes

Additional context I digged a bit into the issue. It seems that creating the 3 CloudFormation stacks takes around 9.5 minutes. The timeout of the lambda and the step related to CreateOrUpdateBaseStack is set to 600s (10 minutes).

Possible workarounds

  • increase timeout of lambda and step to 900s (15minutes)
  • use step MapState with maxConcurency set to 0 for every region (loop over all target regions) for creating the cloudformation stacks
  • remove the option to set the target accounts within the SAR template (cloudformation template) and let the user have another step to update the CodeCommit repository.

AndreasAugustin avatar Jul 06 '21 07:07 AndreasAugustin

Question:

set 3 DeploymentAccountTargetRegions

Are you hosting your deployment pipelines in 3 regions in the deployment account?

Input for a possible solution, why is a Lambda kept alive to track the deployment? This is something you put in a Step Function in a loop. As long as a stack is still being deployed sleep for a minute and retry until either a failure happened or a successful deployment. Then continue the step function.

This would remove the timeout on the Lambda level and reduces the costs of running the lambda.

Nr18 avatar Feb 03 '22 09:02 Nr18

Are you hosting your deployment pipelines in 3 regions in the deployment account?

The deployment pipelines are hosted in a single region. The issue is about the base bootstrapping process. Within the adfconfig.yml you must specify additional regions you like to deploy to. https://github.com/awslabs/aws-deployment-framework/blob/master/docs/admin-guide.md#regions

The issue is about the super first bootstrap process. You receive a timeout if configuring more than one target region.

AndreasAugustin avatar Feb 03 '22 09:02 AndreasAugustin

Thank you for your patience. I am happy to inform you that this issue has been resolved in our latest release v3.2.0 just now. I'm hereby closing this issue. Please open a new issue if you are experiencing any issues with the latest release.

sbkok avatar Jan 24 '23 10:01 sbkok