amplify-category-api icon indicating copy to clipboard operation
amplify-category-api copied to clipboard

UPDATE_ROLLBACK_FAILED state and can not be updated.

Open rafaelfaria opened this issue 1 year ago • 10 comments

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

No response

Amplify CLI Version

12.10.1

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Describe the bug

I am not being able to deploy my project through CLI anymore.

when I try to deploy it now I get the following:

Stack:arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752/a3eccb60-bbba-11ee-9615-0a268650abe7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.

Expected behavior

To push it properly.

Reproduction steps

I am not quite sure how I got to this state thou. However, the only thing I was fixing was that I had a custom Mutation pointing to the wrong lambda function

type Mutation {

  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @function(name: "GetCaptcha-${env}")
    @auth(rules: [{ allow: private, provider: iam }])
}

The following was wrong, since the lambda function was called GenerateCaptcha @function(name: "GetCaptcha-${env}")

When I tried to fix it with the correct name, everything stop working.

Now, it's hard to pinpoint that this was the problem, I got issues with "legacy" lamda layer references in my project, which caused issues too, not sure how this happens.

anyways, hard to pinpoint to something.

I am interested in how I can fix it. I have literally restarted my project and recreated EVERYTHING 3 times during my development due issues. I dont really want to do it again.

Project Identifier

No response

Log output

# Put your logs below this line


Additional information

No response

Before submitting, please confirm:

  • [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
  • [X] I have removed any sensitive information from my code snippets and submission.

rafaelfaria avatar Feb 05 '24 14:02 rafaelfaria

Looking at the stack I found this..

Embedded stack arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752-apiGraphQLAPI-44SP06RETH3B/07ba2540-c1c2-11ee-8d28-0a6786933c53 was not successfully updated. Currently in UPDATE_ROLLBACK_FAILED with reason: The following resource(s) failed to update: [FunctionDirectiveStack].

rafaelfaria avatar Feb 05 '24 14:02 rafaelfaria

image

rafaelfaria avatar Feb 05 '24 14:02 rafaelfaria

So to recap

i managed to deploy the wrong version

  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @function(name: "GetCaptcha-${env}")
    @auth(rules: [{ allow: private, provider: iam }])

Then i tried to fix the function name


  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @function(name: "GenerateCaptcha-${env}")
    @auth(rules: [{ allow: private, provider: iam }])

and that's when i started getting this issue.

image

rafaelfaria avatar Feb 05 '24 14:02 rafaelfaria

Hey,👋 thanks for raising this! I'm going to transfer this over to our API repository for better assistance 🙂

ykethan avatar Feb 05 '24 16:02 ykethan

Looks like the issue is caused by cross stack references. Could you try the below?

  1. Deploy the current state.
  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @function(name: "GetCaptcha-${env}")
    @auth(rules: [{ allow: private, provider: iam }])
  1. Remove the function directive and deploy.
  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @auth(rules: [{ allow: private, provider: iam }])
  1. Deploy with the correct function name.
  generateCaptcha(input: GennerateCaptchaInput): GenerateCaptchaResult
    @function(name: "GenerateCaptcha-${env}")
    @auth(rules: [{ allow: private, provider: iam }])

sundersc avatar Feb 05 '24 19:02 sundersc

@sundersc Thanks for your reply.

Unfortunately It doesn't allow me to redeploy.

is in UPDATE_ROLLBACK_FAILED state and can not be updated.

Every time I try to redeploy, this happens. It doesn't allow me to do anything from the CLI. I'm pretty much stuck and delayed on my project now.

Deploying resources into prod environment. This will take a few minutes. ⠹
Deploying root stack myapp [ ---------------------------------------- ] 0/17
Deploying api GraphQLAPI [ ---------------------------------------- ] 0/20
Deploying function myappAppLayer [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthCreateAuthChallenge [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthDefineAuthChallenge [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthPostConfirmation [ ---------------------------------------- ] 0/4
Deploying function MyAppAuthPreSignup [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthVerifyAuthChallengeResponse [ ---------------------------------------- ] 0/3
Deploying function MyAppCleanUpCaptcha [ ---------------------------------------- ] 0/6
Deploying function MyAppGenerateCaptcha [ ---------------------------------------- ] 0/4
🛑 Stack:arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752/a3eccb60-bbba-11ee-9615-0a268650abe7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

DeploymentFault: Stack:arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752/a3eccb60-bbba-11ee-9615-0a268650abe7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.
    at Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/push-resources.js:362:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:137:16
    at async Promise.all (index 0)
    at async providersPush (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:133:5)
    at async AmplifyToolkit.pushResources (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/push-resources.js:107:13)
    at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
    at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
    at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

Stack:arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752/a3eccb60-bbba-11ee-9615-0a268650abe7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.
ValidationError: Stack:arn:aws:cloudformation:us-east-1:533267446078:stack/amplify-myapp-prod-54752/a3eccb60-bbba-11ee-9615-0a268650abe7 is in UPDATE_ROLLBACK_FAILED state and can not be updated.
    at Request.extractError (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/snapshot/amplify-cli/build/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /snapshot/amplify-cli/build/node_modules/aws-sdk/lib/state_machine.js:26:10
Deploying resources into prod environment. This will take a few minutes. ⠸
Deploying root stack myapp [ ---------------------------------------- ] 0/17
Deploying api GraphQLAPI [ ---------------------------------------- ] 0/20
Deploying function myappAppLayer [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthCreateAuthChallenge [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthDefineAuthChallenge [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthPostConfirmation [ ---------------------------------------- ] 0/4
Deploying function MyAppAuthPreSignup [ ---------------------------------------- ] 0/3
Deploying function MyAppAuthVerifyAuthChallengeResponse [ ---------------------------------------- ] 0/3
Deploying function MyAppCleanUpCaptcha [ ---------------------------------------- ] 0/6
Deploying function MyAppGenerateCaptcha [ ---------------------------------------- ] 0/4

Session Identifier: 8b8724c6-ad64-4045-b54b-559c75011691

It looks like its missing a resolver. Can I manually recreate these resolver (and how?) and rerun the rollback? image

I'm uncertain about the next steps to take. Is there a guide or runbook available for resolving this issue without the risk of losing my work?

This topic has been discussed in other threads, yet it seems there was no conclusive solution provided for when situations reach this point, particularly concerning CLI limitations that can solve this issue. Most discussions seem to end without a clear resolution. I'm hoping to find a way through this and perhaps be the exception by successfully addressing the issue.

I've come across suggestions to delete the cloud stack, establish a new environment, remove the production environment, and then execute amplify init. Would any of these approach be effective? Ex: This issue

rafaelfaria avatar Feb 05 '24 22:02 rafaelfaria

@sundersc Any guidance on this matter? It's been a couple days that I am completely blocked.

rafaelfaria avatar Feb 08 '24 00:02 rafaelfaria

I can't wait any longer, sorry. My app isn't up yet, so losing my data is okay for now. That's why I chose to delete and then set up my environment again, and it worked.

This fix wouldn't work if my app was live. It's a big risk for what I plan to do next. I've been using Amplify for a few years, and it's been tough. I've had to start over many times. Now, I'm thinking about whether to use Amplify for my next project.

I know Amplify is used for many things, but I've seen others have this problem too, without finding a solution or a guide to help. If this had happened when my app was live, it would have been a big problem.

Despite me solving my own problem, I know others will still encounter this in the future. So I hope it doesn't just get ignored and get this thread closed. I don't want to be just another case where no one really tries to find a solution that doesn't mean starting from scratch.

rafaelfaria avatar Feb 08 '24 02:02 rafaelfaria

Of course this is still a problem that the developers of amplify know and won't fix..

AleksandarGT avatar Mar 07 '24 16:03 AleksandarGT

Is there any update ?

fistofzen avatar Aug 23 '24 17:08 fistofzen