aws-toolkit-azure-devops icon indicating copy to clipboard operation
aws-toolkit-azure-devops copied to clipboard

Execute Changeset appears to only work once

Open mcollis2 opened this issue 6 years ago • 4 comments

The first time execute changeset runs, it succeeds. The change is made and the task in Azure Devops completes.

Subsequent executions of the task do apply the changeset, but the task will never complete and has to be cancelled. I believe the reason is that the task is waiting for the stack to reach "create complete" status. However this never happens because thestack status enters "Update Complete" status instead following the initial deployment.

mcollis2 avatar Apr 02 '19 02:04 mcollis2

is this AWS CloudFormation Create/Update Stack task with execute change-set or AWS CloudFormation Execute Change Set task ?

ghost avatar Apr 03 '19 20:04 ghost

The ExecuteChangeSet task should switch to different status waiters (create_complete, update_complete) based on whether the stack has resources at the time the task runs. In the log output, when you're doing an update, are you not seeing the text 'Waiting for stack stackname to reach update complete status'? If you're always seeing 'Waiting for stack stackname to reach create complete status' that would suggest the detection logic is failing for some reason.

Waiters (in the underlying sdk the tasks use) can only wait on one state, hence we have to detect-and-switch.

stevejroberts avatar Apr 03 '19 22:04 stevejroberts

So I think that the logic must be failing. The execute changeset task works great the first time - here is the log for that: 2019-04-03T21:05:22.2061497Z Executing change set build-4-sbox, associated with stack mystack 2019-04-03T21:05:22.4094880Z Waiting for stack mystack to reach create complete status 2019-04-03T21:08:53.7647365Z Execution of change set to create stack mystack completed successfully 2019-04-03T21:08:53.7647811Z Executed change set build-4-sbox 2019-04-03T21:08:53.7729252Z ##[section]Finishing: Execute Change Set: build-4-sbox

When the next run happens, we get this: 2019-04-04T14:47:49.4540941Z Executing change set build-6-sbox, associated with stack mystack2019-04-04T14:47:49.7142750Z Waiting for stack mystack to reach create complete status 2019-04-04T14:50:04.9664798Z ##[error]The operation was canceled. 2019-04-04T14:50:04.9680950Z ##[section]Finishing: Execute Change Set: build-6-sbox

I cancel it because it never completes. Right now I have a powershell task instead that runs and watches for the update. The logic isn't perfect but the task completes and lets us move forwards.

One additional point I would mention is it would also be great to integrate logic that checks a changeset actually exists. When running ci/cd pipelines we may want to trigger them nightly to make sure we don't have drift. If that happens and there are no stack updates, the create/update stack task will pass successfully, but not create a changeset, leading to failure of the subsequent task. This is less of a big deal, but feel like the task should still pass with a log output that no changesets were detected.

Let me know if any further detail is needed!

Thanks, Matthew

mcollis2 avatar Apr 04 '19 14:04 mcollis2

Was this ever resolved? Facing this same exact issue currently.

Creating UPDATE type change set <CHANGESETNAME> Waiting for change set <CHANGESETNAME> to be validated for stack <STACKNAME> Change set successfully validated Executing change set <CHANGESETNAME> <STACKNAME> Waiting for stack <STACKNAME> to reach create complete status

It seems to be waiting for "create complete" when the original type is "UPDATE"

Task definition:

- task: CloudFormationCreateOrUpdateStack@1 displayName: Create and Execute ChangeSet inputs: awsCredentials: <service connection> regionName: <region> stackName: <STACKNAME> templateSource: 'file' templateFile: <template file> templateParametersSource: 'inline' templateParameters: | [ <some parameters> ] useChangeSet: true changeSetName: <CHANGESETNAME> capabilityIAM: false capabilityNamedIAM: false tags: | <some tags>

Deiiv avatar Sep 24 '20 01:09 Deiiv