amplify-category-api icon indicating copy to clipboard operation
amplify-category-api copied to clipboard

Cloudformation Stack stuck in UPDATE_ROLLBACK_FAILED due to ConnectionStack resource failure

Open alex-griffiths opened this issue 1 year ago • 4 comments

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

22.0.0

Amplify CLI Version

12.10.10

What operating system are you using?

Windows

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

I have just tried to push a change to my development environment. The change was simply adding an Auth group to an existing table.

type Asset @model @auth(rules: [
  {allow: private, provider: iam},
  {allow: groups, groupsField: "admin_groups"},
 + {allow: groups, groupsField: "user_groups", operations: [read]}
  ]){
    id: ID!
...
   channel: Channel @hasOne
   admin_groups: [String!]
+  user_groups: [String!]
}

type Channel @model @auth(rules: [
  {allow: private, provider: iam}
]) {
  id: ID!
...
  assetID: ID @index(name: "byChannelAsset", queryField: "getChannelByAsset")
  asset: Asset @belongsTo(fields: ["assetID"])
}

Pushing this up resulted in my Cloudformation stack getting stuck in the UPDATE_ROLLBACK_FAILED state, with various nested stacks in the same state, or some variation of UPDATE_ROLLBACK....

The offending stack appears to be my ConnectionStack which failed with the status "The following resource(s) failed to update: [EFORassetResolver, ChannelassetResolver, DORItemassetResolver, DeviceEngineCycleassetResolver].". Digging into this, the ChannelassetResolver failed with a not found error:

Resource  handler returned message: "No resolver found. (Service: AppSync, Status  Code: 404, Request ID: f0030f5b-3c5c-4fac-9d74-d644bba0a4cc)"  (RequestToken: 149f909d-3ce5-9070-811b-953c45559b91, HandlerErrorCode:  NotFound)

I've previously run into issues with the Cloudformation stacks, and the advice those times has usually been along the lines of, manually rollback the root resources, while skipping the problematic stack, and then fix that stack somehow because the rest of the Cloudformation stack expects it to be a certain way when it is not. I'm not prepared to go through that currently, and would instead like some help understanding first how I got into this mess before trying to make those changes.

Expected behavior

That the Cloudformation stack doesn't getting deadlocked when pushing a change

Reproduction steps

  1. Add auth group to existing table
  2. Push changes with amplify push -y

Project Identifier

90ee5db58b8b8ec82b9721ec8bcea978

Log output

# Put your logs below this line


Additional information

No response

Before submitting, please confirm:

  • [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
  • [X] I have removed any sensitive information from my code snippets and submission.

alex-griffiths avatar Aug 07 '24 06:08 alex-griffiths