Waiter encountered a terminal failure state

[kritro]

Hello Our aws-deployment-framework-pipelines pipeline is working, but still its in a failed state. The error is this 2022-02-08 07:43:48,646 | INFO | s3 | Uploading cdk.out/PIPELINENAME.template.json as pipelines/PIPELINENAME/global.yml to S3 Bucket adf-global-base-deployment-pipelinebucket-xxxxxxx in eu-central-1 | (s3.py:141) Traceback (most recent call last): File "adf-build/cdk/execute_pipeline_stacks.py", line 92, in main() File "adf-build/cdk/execute_pipeline_stacks.py", line 89, in main thread.join() File "/codebuild/output/src973127312/src/adf-build/python/thread.py", line 30, in join raise self.exc File "/codebuild/output/src973127312/src/adf-build/python/thread.py", line 20, in run self.ret = self._target( File "adf-build/cdk/execute_pipeline_stacks.py", line 60, in worker_thread cloudformation.create_stack() File "/codebuild/output/src973127312/src/adf-build/python/cloudformation.py", line 279, in create_stack self._execute_change_set(waiter) File "/codebuild/output/src973127312/src/adf-build/python/cloudformation.py", line 259, in _execute_change_set self._wait_stack(waiter, self.stack_name) File "/codebuild/output/src973127312/src/adf-build/python/cloudformation.py", line 139, in _wait_stack waiter.wait( File "/codebuild/output/src973127312/src/adf-build/botocore/waiter.py", line 53, in wait Waiter.wait(self, **kwargs) File "/codebuild/output/src973127312/src/adf-build/botocore/waiter.py", line 320, in wait raise WaiterError( botocore.exceptions.WaiterError: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

but I cant find any problems on any of the related stacks, either in the deployment account, or the resource account. Does anyone know what this error mean? Trond

[sbkok]

Hi @kritro,

The execute_pipeline_stacks.py script is responsible for creating the deployment pipeline stacks. I see you mentioned that there are no problems in any of the stacks in the deployment account.

Unfortunately, it seems that the script is stuck with one pipeline being in a final state. That means that the CloudFormation template is stuck at a state like:

• CREATE_FAILED,
• ROLLBACK_FAILED,
• DELETE_FAILED,
• UPDATE_ROLLBACK_FAILED, or
• ROLLBACK_COMPLETE.

1. If it is present in a final state that cannot be updated, feel free to delete the stack. In the worst case, the pipeline will be unavailable until the it is generated by ADF again. Resources deployed by the pipeline in a target are not deleted this way. Only the pipeline itself.

2. If that is not the case, can you check the specific pipeline stack and look at the change set that it has. That one might be corrupt/broken. You can safely delete the change set if there is one.

After that, trigger the aws-deployment-framework-pipelines pipeline in CodePipeline so it tries again.

I hope this helps.

[sbkok]

I'm closing this issue as it has been inactive for a few months. This probably means that it is not reproducible or it has been fixed in the meanwhile.

Please reopen if you still encounter this issue with the latest stable version.

Thank you!