Amplify push got error "Message: Resource is not in the state stackUpdateComplete"

[lehoai]

Before opening, please confirm:

• [X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

No response

Amplify CLI Version

Using the latest version at amplify CI/CD

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Amplify Categories

Not applicable

Amplify Commands

push

Describe the bug

I am using CI/CD which links with my GitHub master branch. The last few days ago, it work properly. But now when I try to merge source to master branch, I got the error: [WARNING]: ✖ An error occurred when pushing the resources to the cloud [WARNING]: ✖ There was an error initializing your environment. [INFO]: DeploymentError: ["Index: 1 State: {\"deploy\":\"waitingForDeployment\"} Message: Resource is not in the state stackUpdateComplete"] at /root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/src/iterative-deployment/deployment-manager.ts:159:40 at Interpreter.update (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/interpreter.js:267:9) at /root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/interpreter.js:112:15 at Scheduler.process (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/scheduler.js:69:7) at Scheduler.flushEvents (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/scheduler.js:60:12) at Scheduler.schedule (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/scheduler.js:49:10) at Interpreter.send (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/interpreter.js:106:23) at _a.id (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/xstate/lib/interpreter.js:1017:15) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:95:5)

Then I try with amplify CLI, get the same error too.

Expected behavior

push success.

Reproduction steps

I add a @connection, a @key, and few @aws_subscribe, then push

GraphQL schema(s)

# Put schemas below this line

Log output

# Put your logs below this line

Additional information

No response

[akshbhu]

Hi @lehoai

Can you share you gql schema and categories you have added so that I can reproduce it on my end ?

Also can you share the debug logs present here : ~/.amplify/logs/amplify-cli-<issue-date>.log

Also may I know which amplify version you are using?

[batical]

Got the same issue. very problematic, not able to push anything(dev or production-.

My log are 20k line long.

2022-03-11T12:00:38.110Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (1 of 3)"}])
2022-03-11T12:00:38.110Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:00:38.111Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (1 of 3)"}])
2022-03-11T12:00:39.234Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (1 of 3)"}])
2022-03-11T12:04:41.051Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Waiting for DynamoDB indices to be ready"}])
2022-03-11T12:04:44.266Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:04:44.493Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (2 of 3)"}])
2022-03-11T12:04:44.493Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:04:44.494Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (2 of 3)"}])
2022-03-11T12:04:45.634Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (2 of 3)"}])
2022-03-11T12:08:47.462Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Waiting for DynamoDB indices to be ready"}])
2022-03-11T12:09:51.696Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:09:51.920Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (3 of 3)"}])
2022-03-11T12:09:51.921Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:09:51.924Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (3 of 3)"}])
2022-03-11T12:09:52.941Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Deploying (3 of 3)"}])
2022-03-11T12:21:58.236Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:21:58.239Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (1 of 3)"}])
2022-03-11T12:21:58.240Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (2 of 3)"}])
2022-03-11T12:21:58.241Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:21:58.242Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (2 of 3)"}])
2022-03-11T12:21:58.478Z|error : amplify-provider-awscloudformation.deployment-manager.startRolbackFn([{"index":2}])
Error: Cannot start step then the current step is in ROLLING_BACK status.
2022-03-11T12:21:59.401Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (2 of 3)"}])
2022-03-11T12:26:01.228Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (2 of 3)"}])
2022-03-11T12:26:04.433Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:26:04.648Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (3 of 3)"}])
2022-03-11T12:26:04.650Z|info : amplify-provider-awscloudformation.aws-s3.uploadFile.s3.putObject([{"Key":"[***]ment-[***]json","Bucket":"[***]it-[***]ev-[***]161237-[***]ment"}])
2022-03-11T12:26:04.654Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (3 of 3)"}])
2022-03-11T12:26:05.732Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (3 of 3)"}])
2022-03-11T12:30:07.578Z|info : amplify-provider-awscloudformation.deployment-manager.deploy([{"spinner":"Rolling back (3 of 3)"}])

I try using the last version of amplify cli 7.6.23

here is some part of the log with the rollback starting

[ecc7220]

I have the same issue, I pushed a few destructive changes to my GraphQL model and it failed because a token expired during the push.

2022-03-11T12:22:24.363Z|error : amplify-provider-awscloudformation.aws-s3.uploadFile.s3([{"Key":"[***]ment-[***]json","Bucket":"[***]ify-[***]pool-[***]ing-[***]316-[***]ment"}]) ExpiredToken: The provided token has expired. 2022-03-11T12:22:24.363Z|error : amplify-provider-awscloudformation.deployment-manager.startRolbackFn([{"index":2}]) ExpiredToken: The provided token has expired. 2022-03-11T12:22:38.638Z|error : amplify-provider-awscloudformation.deployment-manager.getTableStatus([{"tableName":"[***]er-[***]6sfqmjiikqe-[***]ing"}]) ExpiredTokenException: The security token included in the request is expired I tried it again and got:

2022-03-11T12:24:18.467Z|info : amplify-provider-awscloudformation.deployment-manager.rollback([{"spinner":"Waiting for previous deployment to finish"}]) 2022-03-11T12:24:18.526Z|error : amplify-provider-awscloudformation.deployment-manager.DeploymentManager([{"stateValue":"failed"}]) DeploymentError: ["Index: 3 State: {\"preRollback\":\"previousDeploymentReadyCheck\"} Message: Resource is not in the state stackUpdateComplete"] Then I pulled the lasted env and pushed and got again:

2022-03-11T12:35:34.798Z|info : amplify-provider-awscloudformation.deployment-manager.rollback([{"spinner":"Waiting for previous deployment to finish"}]) 2022-03-11T12:35:34.834Z|error : amplify-provider-awscloudformation.deployment-manager.DeploymentManager([{"stateValue":"failed"}]) DeploymentError: ["Index: 3 State: {\"preRollback\":\"previousDeploymentReadyCheck\"} Message: Resource is not in the state stackUpdateComplete"] How can I solve this, I'm stuck, please provide steps to fix this, even if I need to remove some stuff, I need to get it working today!

My cli version is 7.6.23, using Cloud9 instance. Only some destructive updates to the model, since the last push. Recreation of tables not an issue right now.

[ecc7220]

I did some research and there are tons of "Resource not in state stackUpdateComplete" issues that where never solved but simply closed. The guys have probably all recreated the whole environments.

Related issues containing "stackUpdateComplete" start at aws-amplify/amplify-cli#82 and goes all the way up to aws-amplify/amplify-category-api#95, with still 18 open issues and 149 closed issues including this one (#9925). Drifts in the stacks happen, you need to handle this properly IMHO. In my case it was the long time the update took, which caused a token to expire.

Amazon are you watching this very common ?

This is a no go.

This is clearly a high priority issue that should be solved once and for all times. Recovery from push failures and stack drifts is very essential, this should simply just work all the time, like a file system.

EDIT:

I could find some more hints, if you change more than 2 indexes in a graphql model, the push fails, complaining about that too many parallel indexes where changed or deleted on a single table, I cant find the log about it, it was displayed on the terminal, but I lost it. The amplify log file in ~/amplify/logs/ is too long already, I can't find anything. After that, you will get the message "Resource is not in the state stackUpdateComplete" and you are stuck. The only way to get it back to work is the deletion and recreation of the environment. Steps I did (on your own risk):

1. Save all your data that is on the amplify controlled amplify storage, also archive the whole environment.
2. Backup all data in your database tables, you will need later to import that data back into new tables. In step 5, the env will be delete and all tables and storage s3 buckets will be deleted! Be careful, you are on your own risk.
3. Create a new env with: amplify env add your_new_env_name
4. You will be now in your new env, the code and backend config should still be there. You can see that with an: amplify status
5. Now delete your old env: amplify env remove broken_env_name (on your own risk)
6. Create a new env with the same name, this env will work: amplify env add broken_env_name
7. you should be in the new env now, check with: amplify status
8. Push your backend into the new env, this will take a while...
9. Now it's the time to reimport all your saved data from step 1 and 2.

The whole procedure takes a long time, is there a better or faster way to do it?

EDIT:

I'm very sure now, that this is caused by too many simultaneous dynamodb index updates on the same table, which hits on a dymanodb limit and that the expired token message was somehow only related to the first error. This also explains why some many people encounter this switching from V1 to V2 graphql models.

So we have two issues:

1. A failed push, for what ever reason needs to be recoverable.
2. GraphQL model updates trigger occasionally simultaneous index updates or deletions which abort the running amplify push, leaving you with an unrecoverable state.

[lehoai]

@akshbhu Sorry, i was so busy. I've changed my schema as above, and a few @subscription and @function. @ecc7220 I cant create a new env because this is a production env.

[batical]

I was able to go though, pushing my changes step by step

• pull env --restore
• pushing multiple changes one by one until I found the error. Took me at least 4 hours ...

Issue cames from a @index error, my bad, but at least an error message can be helpful.

[lehoai]

@batical Can u give me more info? The amplify error log is useless at all!

[ecc7220]

@lehoai I don't know how to recover from this error without recreating the env, sorry, but somebody with deeper knowledge needs to help out somehow. The only thing I could find out, is the cause of the error. Which comes from the index issue I mentioned above. So, don't change too much at once in the model, it could lead to a broken env.

[ecc7220]

Issue aws-amplify/amplify-category-api#88 is also related, I had also tried everything, including the deletion of the "deployment-state.json" file in the corresponding amplify stage bucket. The issue described there is very similar. Only that in my case nothing helped.

@lehoai I would give it a try and delete the file and try pushing again, if you are lucky, you can get it back working.

[lehoai]

@ecc7220 Thanks. I will try.

[ecc7220]

@lehoai if this is not working try also the solution for aws-amplify/amplify-category-api#88. This includes modifying the deployment bucket as well. It is a much better solution, as my solution recreating everything. If I have next time a similar issue, I will try the suggested modification in aws-amplify/amplify-category-api#88. Good luck!

[lehoai]

@ecc7220 Thanks, I will try. I promise you this is the last time I work with Amplify. Terrible tool ever: Unstable, slow support, there is a lot of bugs!

[josefaidt]

Hey @lehoai :wave: apologies for the delay! Can you share the CloudFormation errors that are printed prior to receiving the Resource is not in the state stackUpdateComplete message? Typically when we see this error the CloudFormation errors provide additional insight

[alharris-at]

Hi @lehoai, In addition to the schema errors which Josef has mentioned above specifically, could you also provide a bit more data your environment? There are 2 things which we'd like to understand in a bit more details, auth token expiration and it's impact on your deployment, and the contents of the deployment itself.

1. What auth mechanism are you using for your account? i.e. are you using user access/secret keys for a given user, or are you using something like STS to generate short-lived federated tokens?
2. The schema you're starting out with when you kick off a deployment (previous schema).
3. Schema you are attempting to deploy.

This will help us get understanding of the changes being applied during the deployment. We can set up a call if you'd like as well, rather than sharing the schema publicly on GH, you can reach out to [email protected]

[lehoai]

@alharris-at al Sorry for the late reply. Finally, we have to create a new env and redeploy the whole project. Then it worked. The old env was deleted. So I think it's not a problem with schema (i don't create or update any index).

1. I link the amplify with github in aws console, so it automatically re-deploys every time the source code is merged. I don't use access/secret keys 2, 3. as i said, i don't create or update any index, just add a few subscriptions and columns. I can't show the detail of the schema.

I've checked the error log many times, then there is only one reason "Resource is not in the state stackUpdateComplete", no more. ( i know, this error sometimes shows up when other error occurs, but not in my case, only "Resource is not in the state stackUpdateComplete" is thrown).

[alharris-at]

I see, thank you for the update @lehoai, we're going to create a new bug related to force push behavior in the AWS Console, which sounds related to what you're seeing here. Is there anything else specific we can help you out with on this issue?

[josefaidt]

Hey @lehoai :wave: thank you for those details! To clarify, do you have the affected backend files and would you be willing to send us a zip archive to [email protected]? If so, we would like to take a look and see if we are able to reproduce the issue using your backend definition as we have been unable to reproduce this ourselves.

[lehoai]

@josefaidt @alharris-at Thank you for your response. Honestly, I really wanna share the detail of the schema, and backend files but there is an NDA contract so I cant. I gave u everything I can share above, the error log, part of the schema...

I think u should give more detail in the error log then the developers can investigate the cause.

[naingaungphyo]

I run amplify push and also got the same "Resource is not in the state stackUpdateComplete" error after changing many indexes and the primary key of one of my models. I tried with this amplify/cli/json setting of enableIterativeGsiUpdates as true and also used --force and --allow-destructive-graphql-schema-updates cli flags according to this troubleshooting guide, but none of them works.

By the way, I didn't do any manual modification. (eg. from console etc)

My workaround was removing the api and adding api again as below.

backup my schema file amplify remove api amplify push amplify add api and use my backup schema amplify push used cli version is 8.0.2 and v2 transformer

[josefaidt]

Hey @lehoai no worries on sending the schema. We will continue to investigate this issue.

@naingaungphyo are you seeing any CloudFormation errors outputted to your terminal and would you mind sharing the logs at ~/.amplify/logs the day this occurred? And finally, approximately how many changes were applied prior to receiving this error?

[naingaungphyo]

@josefaidt I added two bidirectional One-to-Many relationship indexes, changed the primary key of a model and added a new GSI.

One thing I forgot to mention is that I added amplify add storage together with the above changes and tried to amplify push all of them at once. Later, I removed both storage and api then added them back one by one with push on each step.

I attached my log starting from adding storage, ending at the error. I masked some information. log.txt

[josefaidt]

Hey @naingaungphyo thanks for the clarification and for posting your logs! I'm taking a further look at this 🙂

[Tony-OAA]

Same Error here!!!

Started on dev environment after updating CLI last week. We had to delete dev env and rebuild.

tried pushing to prod today - same error!

["Index: 1 State: {\"deploy\":\"waitingForDeployment\"} Message: Resource is not in the state stackUpdateComplete"]

This is a nightmare, wasted a whole day on this so far trying to resolve it. Can you please add better error log information that might give us a clue.??? The fact the whole stack gets rebuilt on even a tiny change means a long wait and reams of useless log files to go through. - I don't want to be an expert in debugging cloudformation - Isn't that the point of amplify.

[sachscode]

@naingaungphyo Could you please share in add storage did you deploy S3 bucket or DynamoDB table.

[naingaungphyo]

@sachscode

did you deploy S3 bucket or DynamoDB table

I tried to add s3 bucket and used amplify push to deploy. But it failed.

[jakejcheng]

I added a new model and a relationship in the Studio but then after doing amplify pull, and tried auto-deploy via Github or amplify push, now I'm just getting the "Resource is not in the state stackUpdateComplete" error as well

[jakejcheng]

I run amplify push and also got the same "Resource is not in the state stackUpdateComplete" error after changing many indexes and the primary key of one of my models. I tried with this amplify/cli/json setting of enableIterativeGsiUpdates as true and also used --force and --allow-destructive-graphql-schema-updates cli flags according to this troubleshooting guide, but none of them works.

By the way, I didn't do any manual modification. (eg. from console etc)

My workaround was removing the api and adding api again as below.

backup my schema file amplify remove api amplify push amplify add api and use my backup schema amplify push used cli version is 8.0.2 and v2 transformer

i tried this but I'm still getting the "ResourceNotReady: Resource is not in the state stackUpdateComplete" error on the resource "UpdateRolesWithIDPFunctin"

[naingaungphyo]

@jakejcheng I think that there are some multiple updates at once. (example, updating roles and api both at once) Generally speaking, it should work if you update one by one and amplify push them after each update.

[jakejcheng]

@jakejcheng I think that there are some multiple updates at once. (example, updating roles and api both at once) Generally speaking, it should work if you update one by one and amplify push them after each update.

@naingaungphyo I didn't touch the roles/auth though. I only created a new model and changed an existing model in the console, did a amplify pull, and can no longer successfully push anymore. I tried many solutions from multiple closed/open issues dating back to 2019 but no avail. I was able to push last night before I made a change to the schema in the console. This is extremely frustrating.

[josefaidt]

Hey @jakejcheng would you mind using the CloudFormation console to detect drift in your app's stack? It is recommended to not make changes in the console as that can potentially create drift. Are you continuously experiencing this issue?