Cloudformation waitForStackCreation error

[ansariwn]

Hi,

We use AWS Cloudformation Create/Update VSTS task to create a stack. The VSTS task waits for the stack status to become "Create_Complete". However, we have noticed that our stack status ends up as "Update_Complete" after becoming "Create_Complete" for a short while (less than 30 seconds).

As a result of the above, the "waitForStackCreation" method throws an exception as the method checks for the stack status every 30 seconds and it misses the stack status that is only there for a short while before becoming "Update_Complete". Please see the stack events below, you will notice that the "Create_Complete" status for AWS::Cloudformation:Stack type changes quickly to "Update_Complete".

28/08/2018	Status	Type	Logical ID	Status Reason
	13:08:51 UTC+0100	UPDATE_COMPLETE	AWS::CloudFormation::Stack	licensingserver-533
	13:08:50 UTC+0100	UPDATE_COMPLETE_CLEANUP_IN_PROGRESS	AWS::CloudFormation::Stack	licensingserver-533
	13:08:48 UTC+0100	UPDATE_COMPLETE	AWS::EC2::Instance	EC2Instance
	13:08:31 UTC+0100	UPDATE_IN_PROGRESS	AWS::EC2::Instance	EC2Instance
	13:08:26 UTC+0100	UPDATE_COMPLETE	AWS::EC2::Volume	NewVolume
	13:08:11 UTC+0100	UPDATE_IN_PROGRESS	AWS::EC2::Volume	NewVolume
	13:07:58 UTC+0100	UPDATE_IN_PROGRESS	AWS::CloudFormation::Stack	licensingserver-533
	13:07:53 UTC+0100	CREATE_COMPLETE	AWS::CloudFormation::Stack	licensingserver-533
	13:07:50 UTC+0100	CREATE_COMPLETE	AWS::CloudFormation::WaitCondition	EC2InstanceWaitCondition
	12:37:38 UTC+0100	CREATE_IN_PROGRESS	AWS::CloudFormation::WaitCondition	EC2InstanceWaitCondition
	12:37:38 UTC+0100	CREATE_IN_PROGRESS	AWS::CloudFormation::WaitCondition	EC2InstanceWaitCondition
	12:37:34 UTC+0100	CREATE_COMPLETE	AWS::EC2::Instance	EC2Instance
	12:36:54 UTC+0100	CREATE_IN_PROGRESS	AWS::EC2::Instance	EC2Instance
	12:36:53 UTC+0100	CREATE_IN_PROGRESS	AWS::EC2::Instance	EC2Instance
	12:36:49 UTC+0100	CREATE_COMPLETE	AWS::EC2::Volume	NewVolume
	12:36:37 UTC+0100	CREATE_COMPLETE	Custom::AMIInfo	AMIInfo
	12:36:37 UTC+0100	CREATE_IN_PROGRESS	Custom::AMIInfo	AMIInfo
	12:36:33 UTC+0100	CREATE_IN_PROGRESS	AWS::EC2::Volume	NewVolume
	12:36:32 UTC+0100	CREATE_IN_PROGRESS	AWS::EC2::Volume	NewVolume
	12:36:32 UTC+0100	CREATE_IN_PROGRESS	Custom::AMIInfo	AMIInfo
	12:36:31 UTC+0100	CREATE_COMPLETE	AWS::CloudFormation::WaitConditionHandle	EC2InstanceWaitConditionHandle
	12:36:31 UTC+0100	CREATE_IN_PROGRESS	AWS::CloudFormation::WaitConditionHandle	EC2InstanceWaitConditionHandle
	12:36:31 UTC+0100	CREATE_IN_PROGRESS	AWS::CloudFormation::WaitConditionHandle	EC2InstanceWaitConditionHandle
	12:36:28 UTC+0100	CREATE_IN_PROGRESS	AWS::CloudFormation::Stack	licensingserver-533

[stevejroberts]

I don't know if the waiter (which is implemented in the AWS Javascript SDK) can be customized to look for both states (the documentation seems to indicate it handles only one state at a time). I'll follow up with the sdk team to see what our options are here.

Do we have any insight into what is triggering the update cycle post-completion?

[ansariwn]

Hi @steveataws , thanks for the reply.

I have been investigating what could be triggering the update cycle but no luck so far. Our Cloudformation template looks standard (though I am no expert) where it provisions an EC2Instance and executes some UserData commands successfully. I will keep looking for the reason.

[stevejroberts]

I did check with the sdk team and currently the waiter implementation that's watching for the stack to reach completed status doesn't support waiting on multiple states. The only way it would seem for us to handle a create-complete => update-complete transition would be to abandon use of the built-in sdk waiter and code our own loop to watch the updates.

[Jc692488]

Help

[ansariwn]

@steveataws - The reason for Update_Complete stack status is because of a Lambda function that executes post-completion to apply some tags on the stack. This function is organisation wide and we can't avoid it.

Is it possible to amend the waitForStackCreation function to handle this scenario? Would you be able to help?

[stevejroberts]

It appears that because the sdk waiter implementation cannot wait on multiple states, we'll need to rework that code to abandon the waiter and write our own polling loop. Not a big code change, but various engineers are assigned elsewhere at the moment so I can't promise a fast resolution to the issue right now but I'll see what I can arrange.

[troyattaxact]

What is the status of the feature request? We randomly get the error below while the CloudFromation stack is running and eventually completes successfully in AWS.

Error: Stack xxxxxx failed to reach completion status, error: 'Resource is not in the state stackCreateComplete'