`amplify env checkout` Hangs

[brianlenz]

Before opening, please confirm:

• [X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm -g i @aws-amplify/cli

If applicable, what version of Node.js are you using?

18.7.0

Amplify CLI Version

9.2.1

What operating system are you using?

macOS 12.5.1

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No.

Amplify Categories

auth, storage, function, api, analytics, hosting, notifications

Amplify Commands

env

Describe the bug

We had previously (a few days ago) deployed Hosting to our dev environment. Everything was working great.

Today, we went to deploy Hosting to our production environment. Coincidentally (before we did anything), we happened to notice that the Amplify Hosting environment build had failed 3 minutes before. This was surprising since 7 minutes prior to that, the build worked (and nothing in the Amplify-deployed Next.js code had changed). Upon further inspection, here's the difference in the Backend Build process:

Successful Build

                                 ## Checking for associated backend environment...
                                 ## Backend environment association found
2022-08-30T18:45:58.137Z [INFO]: ## Running amplify pull to generate aws-exports.js file for frontend
2022-08-30T18:46:00.214Z [INFO]: [0mAmplify AppID found: XXXX. Amplify App name is: XXX[0m
2022-08-30T18:46:00.313Z [INFO]: [0mBackend environment dev found in Amplify Console app: XXX[0m
2022-08-30T18:46:01.408Z [WARNING]: - Fetching updates to backend environment: dev from the cloud.
2022-08-30T18:46:02.663Z [INFO]: Overrides functionality is not implemented for this category
                                 Overrides functionality is not implemented for this category
2022-08-30T18:46:02.663Z [INFO]: Overrides functionality is not implemented for this category
2022-08-30T18:46:02.663Z [WARNING]: - Building resource auth/XXX
2022-08-30T18:46:02.879Z [WARNING]: - Building resource auth/userPoolGroups
2022-08-30T18:46:03.027Z [WARNING]: - Building resource api/XXX
2022-08-30T18:46:03.028Z [WARNING]: - Building resource storage/XXX
2022-08-30T18:46:03.028Z [INFO]: Overrides functionality is not implemented for this category
2022-08-30T18:46:03.029Z [INFO]: Overrides functionality is not implemented for this category
                                 Overrides functionality is not implemented for this category
2022-08-30T18:46:03.029Z [WARNING]: ✔ Successfully pulled backend environment dev from the cloud.
2022-08-30T18:46:03.039Z [WARNING]: - Retrieving channel information for APNS.
2022-08-30T18:46:03.117Z [WARNING]: ✔ Channel is not setup for APNS
2022-08-30T18:46:03.117Z [WARNING]: - Retrieving channel information for FCM.
2022-08-30T18:46:03.223Z [WARNING]: ✔ Channel is not setup for FCM
2022-08-30T18:46:03.223Z [WARNING]: - Retrieving channel information for Email.
2022-08-30T18:46:03.306Z [WARNING]: ✔ Channel is not setup for Email
2022-08-30T18:46:03.306Z [WARNING]: - Retrieving channel information for SMS.
2022-08-30T18:46:03.362Z [WARNING]: ✔ Channel is not setup for SMS
2022-08-30T18:46:03.657Z [INFO]: ✅
2022-08-30T18:46:04.314Z [WARNING]: - Uploading files...

Failed Build

This happened 7 minutes later, without any changes to Amplify:

                                 ## Checking for associated backend environment...
                                 ## Backend environment association found
2022-08-30T19:47:29.419Z [INFO]: ## Running amplify pull to generate aws-exports.js file for frontend
2022-08-30T19:47:31.488Z [INFO]: [0mAmplify AppID found: XXX. Amplify App name is: XXX[0m
2022-08-30T19:47:31.567Z [INFO]: [0mBackend environment dev found in Amplify Console app: XXX[0m

That's the extent of the Backend Build log, so something obviously stopped working.

We then went back to the CLI on a local developer machine and tried to run an amplify env checkout dev. This process now hangs indefinitely (whereas before we could easily switch between environments in about 10-15 seconds):

$ amplify env checkout dev
⠇ Initializing your environment: dev

The same thing happens for an amplify pull:

$ amplify pull --appId XXX --envName dev
⠧ Fetching updates to backend environment: dev from the cloud.

Expected behavior

Amplify Backend Build should successfully check out the dev environment config and place the aws-exports.js as appropriate.

Further, amplify env checkout dev and amplify pull --appId XXX --envName dev should complete successfully in a reasonable amount of time (< 15s).

Reproduction steps

1. amplify env checkout dev OR
2. amplify pull --appId XXX --envName dev OR
3. Trigger a dev build on Amplify console.

GraphQL schema(s)

N/A

Project Identifier

4e21dc38ea308f3725f90a3dbaf649db

Log output

N/A

Additional information

We are using a monorepo with the following structure:

/amplify /app (for a React Native app) /web (for the Amplify-hosted environment - Next.js)

The only other part to add here is that we did an amplify push with our production environment after this issue occurred. The only change for production was to add Hosting. That all worked correctly. The production hosting is working perfectly, but the dev environment remains broken, as described above.

It's also worth noting that we are using Amplify CLI 9.1.0 for Amplify builds on the console in order to work around the "Creating a nested amplify project is not supported." issue (which is currently broken on 9.2.1):

https://github.com/aws-amplify/amplify-cli/issues/10832#issuecomment-1225879252 https://github.com/aws-amplify/amplify-hosting/issues/2908

[brianlenz]

Since we are using Amplify CLI 9.1.0 for Amplify frontend builds in the console, I tried switching to the 9.1.0 CLI locally, and it has the same hanging issue, so it doesn't seem to matter if we use 9.1.0 or 9.2.1: the environment checkout doesn't ever finish (even after waiting hours while the command runs).

[josefaidt]

Hey @brianlenz :wave: thanks for raising this! To better understand the context of the issue we have a few follow-up questions:

• would you mind posting the latest logs from ~/.amplify/logs?
• are there nested amplify/ directories in either/both of the monorepo apps? (/app/amplify?)
• are you able to pull this project into a fresh directory?

[brianlenz]

@josefaidt, thanks for the response! My follow-ups:

• The log is 2.2MB and contains project-specific info we'd prefer not to post publicly. Is there some way to share it privately?
• No, there aren't any nested amplify directories. The amplify directory lives in the monorepo root, and it's the only one that exists.
• It doesn't look like it:

$ amplify pull --appId XXX --envName dev
? Select the authentication method you want to use: AWS profile

For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

? Please choose the profile you want to use default
Amplify AppID found: XXX. Amplify App name is: YYY
Backend environment dev found in Amplify Console app: YYY
$ amplify status
Name: NotInitializedError : Message: 
      No Amplify backend project files detected within this folder. Either initialize a new Amplify project or pull an existing project.
      - "amplify init" to initialize a new Amplify project
      - "amplify pull <app-id>" to pull your existing Amplify project. Find the <app-id> in the AWS Console or Amplify Admin UI.

It works when I do the pull into a fresh directory for our production environment, though.

[brianlenz]

@josefaidt, I don't know if this is related, but we noticed today that when we view the details of a backend environment in the Amplify console, the page "crashes" and disappears. Starting here:

If we click on dev or View details, it opens the details page:

A few seconds (3-10) after the page has loaded, the contents of the page entirely disappear and it's broken:

This issue happens both for the dev environment and the production environment, which is interesting.

Just thought I would mention it since we noticed it, in case it is in any way related to the underlying issue.

[brianlenz]

@josefaidt, FYI, I successfully created a new staging environment yesterday. It's up and running without issue, as production continues to be. For whatever reason, the dev environment remains broken when attempting to checkout with the CLI. The dev environment itself is functioning just fine (and all of our apps run successfully against it), so the issue appears to be purely with the management of the environment with the Amplify CLI.

It would be great if we had a way to get it fixed and working again. I also think it would be good for the Amplify ecosystem to get the underlying issue fixed so that other users don't run into it. I have a fairly high level of confidence that there's nothing we did that could have broken things. We've been using standard Amplify CLI commands with very little custom behavior in the AWS console, and at the time that it broke, we weren't even making any changes to the environment.

If there's anything I can help with to further the process, let me know! I'd be happy to privately share the Amplify CLI log with you (from the day that things went haywire), as well.

Thanks much!

[josefaidt]

Hey @brianlenz apologies for the delay but I'm glad to hear the new environment created successfully. Given you were able to create a new environment, can you verify the integrity of the #current-cloud-backend folder after pulling into a fresh directory and the amplify/backend/amplify-meta.json file?

The log is 2.2MB and contains project-specific info we'd prefer not to post publicly. Is there some way to share it privately?

You can try emailing them to [email protected]

[josefaidt]

Are there any errors in the browser console when the screen goes blank?

[brianlenz]

Thanks, @josefaidt, and no worries on the delay 👍

I can successfully pull production and staging into an empty directory. Everything looks good for both of those in #current-cloud-backend and amplify/backend/amplify-meta.json. I can switch between them with amplify env checkout and I can do amplify pull, and everything appears to work great.

As soon as I attempt to pull the dev environment, everything goes haywire, and the amplify directories get messed up. When I attempt to pull dev, the pull fails (as shown above). At that point, the #current-cloud-backend directory is empty, as is amplify/backend. So, simply checking out dev seems to break it all.

Yes, there are errors in the browser console when the screen goes blank (redacted some of the tokens and IDs):

vendors~main.js:43          GET https://aws-amplify-prod-us-west-2-artifacts.s3.us-west-2.amazonaws.com/XXXXXX/main/SCREENSHOTS/thumbnail.png?X-Amz-Security-Token=YYYYY%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220907T192539Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Credential=ZZZ%2F20220907%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=QQQQQ 404 (Not Found)
(anonymous) @ vendors~main.js:43
e.exports @ vendors~main.js:43
e.exports @ vendors~main.js:106
Promise.then (async)
s.request @ vendors~main.js:106
i.forEach.s.<computed> @ vendors~main.js:106
(anonymous) @ vendors~main.js:43
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
N @ main.js:1
F.getScreenshot @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
N @ main.js:1
(anonymous) @ main.js:1
F.generateAppSummary @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
N @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
Promise.then (async)
(anonymous) @ main.js:1
F.listApps @ main.js:1
V @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
k @ vendors~main.js:3
z @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
k @ vendors~main.js:3
z @ vendors~main.js:3
V @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
u @ vendors~main.js:3
t @ vendors~main.js:3
u @ vendors~main.js:3
vendors~main.js:59 TypeError: Cannot read properties of undefined (reading 'includes')
    at main.js:1:4067678
    at Array.some (<anonymous>)
    at F.V.generateTabs (main.js:1:4067651)
    at F.render (main.js:1:4069749)
    at zo (vendors~main.js:59:71428)
    at Uo (vendors~main.js:59:71227)
    at zs (vendors~main.js:59:112369)
    at ku (vendors~main.js:59:98325)
    at Cu (vendors~main.js:59:98253)
    at Su (vendors~main.js:59:98116)
as @ vendors~main.js:59
r.callback @ vendors~main.js:59
ca @ vendors~main.js:59
ds @ vendors~main.js:59
Tu @ vendors~main.js:59
t.unstable_runWithPriority @ vendors~main.js:67
zi @ vendors~main.js:59
Iu @ vendors~main.js:59
du @ vendors~main.js:59
(anonymous) @ vendors~main.js:59
t.unstable_runWithPriority @ vendors~main.js:67
zi @ vendors~main.js:59
Vi @ vendors~main.js:59
Gi @ vendors~main.js:59
mu @ vendors~main.js:59
notify @ vendors~main.js:11
notifyNestedSubs @ vendors~main.js:11
i @ vendors~main.js:11
g @ vendors~main.js:11
(anonymous) @ vendors~main.js:3
dispatch @ vendors~main.js:11
(anonymous) @ vendors~main.js:1
(anonymous) @ vendors~main.js:3
a @ vendors~main.js:10
c @ vendors~main.js:10
o @ vendors~main.js:10
G @ vendors~main.js:3
O @ vendors~main.js:3
(anonymous) @ vendors~main.js:3
J @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
u @ vendors~main.js:3
Promise.then (async)
U @ vendors~main.js:3
V @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
k @ vendors~main.js:3
z @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
k @ vendors~main.js:3
z @ vendors~main.js:3
V @ vendors~main.js:3
O @ vendors~main.js:3
F @ vendors~main.js:3
u @ vendors~main.js:3
t @ vendors~main.js:3
u @ vendors~main.js:3
Show 3 more frames
vendors~main.js:1 uncaught at S TypeError: Cannot read properties of undefined (reading 'includes')
    at https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/main.js:1:4067678
    at Array.some (<anonymous>)
    at F.V.generateTabs (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/main.js:1:4067651)
    at F.render (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/main.js:1:4069749)
    at zo (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:71428)
    at Uo (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:71227)
    at zs (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:112369)
    at ku (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:98325)
    at Cu (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:98253)
    at Su (https://a.b.cdn.console.awsstatic.com/11022fed2ea55caa01871e7255e78d89d496ec90731a4fc222d5f8ea4b60e7ce/vendors~main.js:59:98116)

I've emailed the logs to [email protected] 🚀

[biller-aivy]

can you try to open the #current-cloud-backend file in the deployment bucket (s3) for dev env?

[brianlenz]

@biller-aivy sure! I suspect you've hit the heart of the issue. It looks like that file is corrupted. Do you have any ideas how to fix this?

$ unzip \#current-cloud-backend.zip 
Archive:  #current-cloud-backend.zip
  inflating: cli.json                
  inflating: amplify-meta.json       
  error:  invalid compressed data to inflate
file #3:  bad zipfile offset (local header sig):  3096
file #4:  bad zipfile offset (local header sig):  3136
file #5:  bad zipfile offset (local header sig):  3181
file #6:  bad zipfile offset (local header sig):  3231
file #7:  bad zipfile offset (local header sig):  6447
file #8:  bad zipfile offset (local header sig):  6774
file #9:  bad zipfile offset (local header sig):  6829
file #10:  bad zipfile offset (local header sig):  9739
file #11:  bad zipfile offset (local header sig):  10109
file #12:  bad zipfile offset (local header sig):  10143
file #13:  bad zipfile offset (local header sig):  10206
file #14:  bad zipfile offset (local header sig):  10245
file #15:  bad zipfile offset (local header sig):  10634
file #16:  bad zipfile offset (local header sig):  10886
file #17:  bad zipfile offset (local header sig):  10941
file #18:  bad zipfile offset (local header sig):  11225
file #19:  bad zipfile offset (local header sig):  11937
file #20:  bad zipfile offset (local header sig):  12227
file #21:  bad zipfile offset (local header sig):  12942
file #22:  bad zipfile offset (local header sig):  14265
api/ttw3/build/resolvers/Mutation.createJournalMigration.req.vtl:  mismatching "local" filename (api/ttw3/build/resolvers/Mutation.createAudioGuide.res.vtl),
         continuing with "central" filename version
  inflating: api/ttw3/build/resolvers/Mutation.createJournalMigration.req.vtl   bad CRC 92a14e9a  (should be b9a63b49)
file #24:  bad zipfile offset (local header sig):  14681
file #25:  bad zipfile offset (local header sig):  14823
file #26:  bad zipfile offset (local header sig):  16136
file #27:  bad zipfile offset (local header sig):  16306
file #28:  bad zipfile offset (local header sig):  17623
file #29:  bad zipfile offset (local header sig):  17796
file #30:  bad zipfile offset (local header sig):  19365
file #31:  bad zipfile offset (local header sig):  19535
file #32:  bad zipfile offset (local header sig):  21123
file #33:  bad zipfile offset (local header sig):  21307
file #34:  bad zipfile offset (local header sig):  22893
file #35:  bad zipfile offset (local header sig):  23076
file #36:  bad zipfile offset (local header sig):  24620
file #37:  bad zipfile offset (local header sig):  24799
file #38:  bad zipfile offset (local header sig):  26367
file #39:  bad zipfile offset (local header sig):  26541
file #40:  bad zipfile offset (local header sig):  28175
api/ttw3/build/resolvers/Mutation.deletePlan.req.vtl:  mismatching "local" filename (api/ttw3/build/resolvers/Mutation.deleteAudioGuide.res.vtl),
         continuing with "central" filename version
  inflating: api/ttw3/build/resolvers/Mutation.deletePlan.req.vtl   bad CRC 92a14e9a  (should be 8ad980b3)
file #42:  bad zipfile offset (local header sig):  29979
file #43:  bad zipfile offset (local header sig):  30149
file #44:  bad zipfile offset (local header sig):  31780
file #45:  bad zipfile offset (local header sig):  31953
file #46:  bad zipfile offset (local header sig):  33749
file #47:  bad zipfile offset (local header sig):  33919
file #48:  bad zipfile offset (local header sig):  35745
file #49:  bad zipfile offset (local header sig):  35807
file #50:  bad zipfile offset (local header sig):  35991
file #51:  bad zipfile offset (local header sig):  37464
file #52:  bad zipfile offset (local header sig):  39289
file #53:  bad zipfile offset (local header sig):  45826
file #54:  bad zipfile offset (local header sig):  46077
file #55:  bad zipfile offset (local header sig):  46260
file #56:  bad zipfile offset (local header sig):  46316
file #57:  bad zipfile offset (local header sig):  47550
file #58:  bad zipfile offset (local header sig):  49340
file #59:  bad zipfile offset (local header sig):  49501
file #60:  bad zipfile offset (local header sig):  49547
file #61:  bad zipfile offset (local header sig):  49726
file #62:  bad zipfile offset (local header sig):  52081
file #63:  bad zipfile offset (local header sig):  52739
file #64:  bad zipfile offset (local header sig):  54547
file #65:  bad zipfile offset (local header sig):  54673
file #66:  bad zipfile offset (local header sig):  54847
file #67:  bad zipfile offset (local header sig):  54882
file #68:  bad zipfile offset (local header sig):  57220
file #69:  bad zipfile offset (local header sig):  57276
file #70:  bad zipfile offset (local header sig):  57452
file #71:  bad zipfile offset (local header sig):  59760
file #72:  bad zipfile offset (local header sig):  61296
file #73:  bad zipfile offset (local header sig):  61466
file #74:  bad zipfile offset (local header sig):  61516
file #75:  bad zipfile offset (local header sig):  63815
file #76:  bad zipfile offset (local header sig):  63982
file #77:  bad zipfile offset (local header sig):  64155
file #78:  bad zipfile offset (local header sig):  64203
file #79:  bad zipfile offset (local header sig):  66648
file #80:  bad zipfile offset (local header sig):  66702
file #81:  bad zipfile offset (local header sig):  66872
file #82:  bad zipfile offset (local header sig):  66936
file #83:  bad zipfile offset (local header sig):  69407
file #84:  bad zipfile offset (local header sig):  69481
file #85:  bad zipfile offset (local header sig):  69665
file #86:  bad zipfile offset (local header sig):  72599
file #87:  bad zipfile offset (local header sig):  75068
file #88:  bad zipfile offset (local header sig):  75126
file #89:  bad zipfile offset (local header sig):  75309
file #90:  bad zipfile offset (local header sig):  75372
file #91:  bad zipfile offset (local header sig):  77826
file #92:  bad zipfile offset (local header sig):  77895
file #93:  bad zipfile offset (local header sig):  78074
file #94:  bad zipfile offset (local header sig):  81314
file #95:  bad zipfile offset (local header sig):  83767
file #96:  bad zipfile offset (local header sig):  83826
file #97:  bad zipfile offset (local header sig):  84000
file #98:  bad zipfile offset (local header sig):  84080
file #99:  bad zipfile offset (local header sig):  84906
file #100:  bad zipfile offset (local header sig):  84992
file #101:  bad zipfile offset (local header sig):  85774
file #102:  bad zipfile offset (local header sig):  92335
file #103:  bad zipfile offset (local header sig):  92618
file #104:  bad zipfile offset (local header sig):  95712
file #105:  bad zipfile offset (local header sig):  96421
api/ttw3/build/resolvers/Query.getAudioGuide.req.vtl:  mismatching "local" filename (api/ttw3/build/resolvers/Subscription.onUpdatePlan.res.vtl),
         continuing with "central" filename version
  inflating: api/ttw3/build/resolvers/Query.getAudioGuide.req.vtl  
  error:  invalid compressed data to inflate
file #107:  bad zipfile offset (local header sig):  96737
file #108:  bad zipfile offset (local header sig):  96817
file #109:  bad zipfile offset (local header sig):  97533
file #110:  bad zipfile offset (local header sig):  99912
file #111:  bad zipfile offset (local header sig):  100146
file #112:  bad zipfile offset (local header sig):  100218
file #113:  bad zipfile offset (local header sig):  100354
file #114:  bad zipfile offset (local header sig):  100432
file #115:  bad zipfile offset (local header sig):  100668
file #116:  bad zipfile offset (local header sig):  106129
file #117:  bad zipfile offset (local header sig):  106839
file #118:  bad zipfile offset (local header sig):  111199
file #119:  bad zipfile offset (local header sig):  111438
file #120:  bad zipfile offset (local header sig):  111501
file #121:  bad zipfile offset (local header sig):  112214
file #122:  bad zipfile offset (local header sig):  112290
file #123:  bad zipfile offset (local header sig):  112526
file #124:  bad zipfile offset (local header sig):  114295
file #125:  bad zipfile offset (local header sig):  115164
file #126:  bad zipfile offset (local header sig):  115266
file #127:  bad zipfile offset (local header sig):  115516
file #128:  bad zipfile offset (local header sig):  117777
file #129:  bad zipfile offset (local header sig):  118678
file #130:  bad zipfile offset (local header sig):  118778
file #131:  bad zipfile offset (local header sig):  119027
file #132:  bad zipfile offset (local header sig):  120508
file #133:  bad zipfile offset (local header sig):  121407
file #134:  bad zipfile offset (local header sig):  121500
file #135:  bad zipfile offset (local header sig):  121745
file #136:  bad zipfile offset (local header sig):  123244
file #137:  bad zipfile offset (local header sig):  124112
file #138:  bad zipfile offset (local header sig):  124189
file #139:  bad zipfile offset (local header sig):  124429
file #140:  bad zipfile offset (local header sig):  126525
file #141:  bad zipfile offset (local header sig):  127409
file #142:  bad zipfile offset (local header sig):  132846
file #143:  bad zipfile offset (local header sig):  133318
file #144:  bad zipfile offset (local header sig):  133380
file #145:  bad zipfile offset (local header sig):  134149
file #146:  bad zipfile offset (local header sig):  134222
file #147:  bad zipfile offset (local header sig):  134688
file #148:  bad zipfile offset (local header sig):  136768
file #149:  bad zipfile offset (local header sig):  137531
file #150:  bad zipfile offset (local header sig):  141697
file #151:  bad zipfile offset (local header sig):  142166
file #152:  bad zipfile offset (local header sig):  143141
file #153:  bad zipfile offset (local header sig):  143907
file #154:  bad zipfile offset (local header sig):  143946
file #155:  bad zipfile offset (local header sig):  144426
file #156:  bad zipfile offset (local header sig):  144478
file #157:  bad zipfile offset (local header sig):  145450
file #158:  bad zipfile offset (local header sig):  145651
file #159:  bad zipfile offset (local header sig):  146130
file #160:  bad zipfile offset (local header sig):  146352
file #161:  bad zipfile offset (local header sig):  147323
file #162:  bad zipfile offset (local header sig):  147379
file #163:  bad zipfile offset (local header sig):  147854
file #164:  bad zipfile offset (local header sig):  148129
file #165:  bad zipfile offset (local header sig):  149066
file #166:  bad zipfile offset (local header sig):  149128
file #167:  bad zipfile offset (local header sig):  149598
file #168:  bad zipfile offset (local header sig):  149669
file #169:  bad zipfile offset (local header sig):  150624
file #170:  bad zipfile offset (local header sig):  1577708
file #171:  bad zipfile offset (local header sig):  1578174
file #172:  bad zipfile offset (local header sig):  1578335
file #173:  bad zipfile offset (local header sig):  1579273
file #174:  bad zipfile offset (local header sig):  1579394
file #175:  bad zipfile offset (local header sig):  1579552
file #176:  bad zipfile offset (local header sig):  1592334
file #177:  bad zipfile offset (local header sig):  1592996
file #178:  bad zipfile offset (local header sig):  1593310
file #179:  bad zipfile offset (local header sig):  1593462
file #180:  bad zipfile offset (local header sig):  1615330
file #181:  bad zipfile offset (local header sig):  1615986
file #182:  bad zipfile offset (local header sig):  1617731
file #183:  bad zipfile offset (local header sig):  1617886
file #184:  bad zipfile offset (local header sig):  1617962
file #185:  bad zipfile offset (local header sig):  1618621
file #186:  bad zipfile offset (local header sig):  1618846
file #187:  bad zipfile offset (local header sig):  1618998
file #188:  bad zipfile offset (local header sig):  1619391
file #189:  bad zipfile offset (local header sig):  1620198
file #190:  bad zipfile offset (local header sig):  1620385
file #191:  bad zipfile offset (local header sig):  1620551
file #192:  bad zipfile offset (local header sig):  1620631
file #193:  bad zipfile offset (local header sig):  1620769
file #194:  bad zipfile offset (local header sig):  1621068
file #195:  bad zipfile offset (local header sig):  1621233
file #196:  bad zipfile offset (local header sig):  1621319
file #197:  bad zipfile offset (local header sig):  1621456
file #198:  bad zipfile offset (local header sig):  1621551
file #199:  bad zipfile offset (local header sig):  1621712
file #200:  bad zipfile offset (local header sig):  3048820
file #201:  bad zipfile offset (local header sig):  3049617
file #202:  bad zipfile offset (local header sig):  3049802
file #203:  bad zipfile offset (local header sig):  3049958
file #204:  bad zipfile offset (local header sig):  3050763
file #205:  bad zipfile offset (local header sig):  3050891
file #206:  bad zipfile offset (local header sig):  3051074
file #207:  bad zipfile offset (local header sig):  3051232
file #208:  bad zipfile offset (local header sig):  3051673
file #209:  bad zipfile offset (local header sig):  3052335
file #210:  bad zipfile offset (local header sig):  3052647
file #211:  bad zipfile offset (local header sig):  3052799
file #212:  bad zipfile offset (local header sig):  3065635
file #213:  bad zipfile offset (local header sig):  3066291
file #214:  bad zipfile offset (local header sig):  3067748
file #215:  bad zipfile offset (local header sig):  3067903
file #216:  bad zipfile offset (local header sig):  3067972
file #217:  bad zipfile offset (local header sig):  3068631
file #218:  bad zipfile offset (local header sig):  3068848
file #219:  bad zipfile offset (local header sig):  3069000
file #220:  bad zipfile offset (local header sig):  3069147
file #221:  bad zipfile offset (local header sig):  3069954
file #222:  bad zipfile offset (local header sig):  3070343
file #223:  bad zipfile offset (local header sig):  3070509
file #224:  bad zipfile offset (local header sig):  3070676
file #225:  bad zipfile offset (local header sig):  3070814
file #226:  bad zipfile offset (local header sig):  3070887
file #227:  bad zipfile offset (local header sig):  3071052
file #228:  bad zipfile offset (local header sig):  3071344
file #229:  bad zipfile offset (local header sig):  3071481
file #230:  bad zipfile offset (local header sig):  3071560
file #231:  bad zipfile offset (local header sig):  3071721
file #232:  bad zipfile offset (local header sig):  3071809
file #233:  bad zipfile offset (local header sig):  3072606
file #234:  bad zipfile offset (local header sig):  4499707
file #235:  bad zipfile offset (local header sig):  4499863
file #236:  bad zipfile offset (local header sig):  4500041
file #237:  bad zipfile offset (local header sig):  4500169
file #238:  bad zipfile offset (local header sig):  4501353
file #239:  bad zipfile offset (local header sig):  4501511
file #240:  bad zipfile offset (local header sig):  4501698
file #241:  bad zipfile offset (local header sig):  4502360
file #242:  bad zipfile offset (local header sig):  4503031
file #243:  bad zipfile offset (local header sig):  4503183
file #244:  bad zipfile offset (local header sig):  4503509
file #245:  bad zipfile offset (local header sig):  4504165
file #246:  bad zipfile offset (local header sig):  4516997
file #247:  bad zipfile offset (local header sig):  4517152
file #248:  bad zipfile offset (local header sig):  4518627
file #249:  bad zipfile offset (local header sig):  4519286
file #250:  bad zipfile offset (local header sig):  4519324
file #251:  bad zipfile offset (local header sig):  4519476
file #252:  bad zipfile offset (local header sig):  4519529
file #253:  bad zipfile offset (local header sig):  4520336
file #254:  bad zipfile offset (local header sig):  4520374
file #255:  bad zipfile offset (local header sig):  4520540
file #256:  bad zipfile offset (local header sig):  4520589
file #257:  bad zipfile offset (local header sig):  4520727
file #258:  bad zipfile offset (local header sig):  4521186
file #259:  bad zipfile offset (local header sig):  4521351
file #260:  bad zipfile offset (local header sig):  4523335
file #261:  bad zipfile offset (local header sig):  4523472
file #262:  bad zipfile offset (local header sig):  4523633
file #263:  bad zipfile offset (local header sig):  4524430
file #264:  bad zipfile offset (local header sig):  4524586
file #265:  bad zipfile offset (local header sig):  4524714
file #266:  bad zipfile offset (local header sig):  4525268
file #267:  bad zipfile offset (local header sig):  4526053
file #268:  bad zipfile offset (local header sig):  4526877
file #269:  bad zipfile offset (local header sig):  4527656
file #270:  bad zipfile offset (local header sig):  4528494
file #271:  bad zipfile offset (local header sig):  4529476
file #272:  bad zipfile offset (local header sig):  4530312
file #273:  bad zipfile offset (local header sig):  4531292
file #274:  bad zipfile offset (local header sig):  4531589
file #275:  bad zipfile offset (local header sig):  4532476
file #276:  bad zipfile offset (local header sig):  4533301
file #277:  bad zipfile offset (local header sig):  4534271
file #278:  bad zipfile offset (local header sig):  4534580
file #279:  bad zipfile offset (local header sig):  4535306
file #280:  bad zipfile offset (local header sig):  4535607
file #281:  bad zipfile offset (local header sig):  4536486
file #282:  bad zipfile offset (local header sig):  4536795
file #283:  bad zipfile offset (local header sig):  4537689
file #284:  bad zipfile offset (local header sig):  4537987
file #285:  bad zipfile offset (local header sig):  4538706
file #286:  bad zipfile offset (local header sig):  4539007
file #287:  bad zipfile offset (local header sig):  4539885
file #288:  bad zipfile offset (local header sig):  4540731
file #289:  bad zipfile offset (local header sig):  4541717
file #290:  bad zipfile offset (local header sig):  4542015
file #291:  bad zipfile offset (local header sig):  4542733
file #292:  bad zipfile offset (local header sig):  4543016
file #293:  bad zipfile offset (local header sig):  4543726
file #294:  bad zipfile offset (local header sig):  4544007
file #295:  bad zipfile offset (local header sig):  4544876
file #296:  bad zipfile offset (local header sig):  4548447
file #297:  bad zipfile offset (local header sig):  4548499
file #298:  bad zipfile offset (local header sig):  4552945
file #299:  bad zipfile offset (local header sig):  4556355
file #300:  bad zipfile offset (local header sig):  4557065
file #301:  bad zipfile offset (local header sig):  4559903
file #302:  bad zipfile offset (local header sig):  4564258
file #303:  bad zipfile offset (local header sig):  4568378
file #304:  bad zipfile offset (local header sig):  4572506
  inflating: api/ttw3/build/stacks/UserCompletedPlan.json  
  inflating: api/ttw3/build/stacks/UserMigration.json  
  inflating: api/ttw3/build/stacks/UserPlan.json  

[josefaidt]

Hey @brianlenz I've added our pull label to track the issue around failed pulls deleting the files that are not tracked by git and leaving you in a state with an amplify-backup directory and an empty amplify/backend directory, which if deployed would delete the backend resources for this environment.

Amplify console, the page "crashes" and disappears

We are tracking this issue here https://github.com/aws-amplify/amplify-hosting/issues/2855

For the corrupted #current-cloud-backend.zip artifact from the original report, I will continue to investigate this with our Hosting team. As a follow up question, is this backend connected to another frontend as well?

[brianlenz]

Thanks, @josefaidt! Right now, the dev environment (backend) is not connected to a frontend. The staging and production backends are the two environments currently connected to frontends.

Thanks for your continued support!

[brianlenz]

@josefaidt I should mention that at one point the dev environment had been connected to a frontend, but we switched things around so that is no longer the case.

[biller-aivy]

@brianlenz had the same case. I added some frontends to my backend app through the amplify console "new app" function. The CLI V. 7.26 was the last working version, after that, amplify console build was always destroy my #current-cloud-backend zip file. The CLI from my local Mac was fine with v8. I just pulled the aws-export.js and don't build the backend within the frontend deploy. I tested now v10 in the build settings, and it is working.

If you have a git repo, you can restore your #current-cloud-backend zip. Just zip your amplify/backend as #current-cloud-backend.zip, important is, that you mark the files itself, and not the backend folder. So that there is no parent folder inside the zip. After that, you can upload it to s3 and it should work again.

[josefaidt]

Hey @biller-aivy thanks for chiming in here with the workaround to restore the current-cloud-backend zip archive, @brianlenz did this work for you?

[brianlenz]

Hi @josefaidt! I haven't had a chance yet to try this. I need to go back in our git history to (approximately) the day that we had this issue and see if I can restore it from that. One question I have is: what could happen if we create current-cloud-backend from data that doesn't exactly represent what is currently deployed? What could go wrong?

Worst case, I will just manually destroy the environment and all its resources and start over creating a new environment from scratch. It's not ideal, but it's workable for us.

Lastly, I personally feel it would be worth figuring out why the Amplify CLI broke the management of our dev environment at all. We didn't do anything specifically that would have corrupted this zip file (and I'd argue the Amplify CLI should never let such a thing occur), so there must be some kind of bug in Amplify CLI that resulted in this issue, right?

[josefaidt]

Hey @brianlenz no worries!

what could happen if we create current-cloud-backend from data that doesn't exactly represent what is currently deployed? What could go wrong?

I tried this out with a sample project and it appears to not have disturbed existing resources, and instead is confused about the resource state for some of the newer resources (ones that have not already been pushed)

While in the affected environment:

1. make a copy of the zip archive in your s3 deployment bucket for good measure
2. in your project, cd into backend directory cd amplify/backend
3. zip contents with zip -r "../../#current-cloud-backend.zip" * (this will create the zip archive in your project root)
4. upload zip contents to your s3 deployment bucket
5. delete the amplify/#current-cloud-backend directory locally
6. run amplify init --envName dev --yes to reinitialize the project (this will pull the contents of the recently uploaded zip archive)
7. run amplify status, observe mismatch in resource states
8. run amplify push --force to forcefully push resources

Lastly, I personally feel it would be worth figuring out why the Amplify CLI broke the management of our dev environment at all. We didn't do anything specifically that would have corrupted this zip file (and I'd argue the Amplify CLI should never let such a thing occur), so there must be some kind of bug in Amplify CLI that resulted in this issue, right?

Great callout, and yes, there is a known instance we are investigating where using the Amplify Hosting feature to connect new frontend apps to existing backends, the backend is undergoing a push when we pushed frontend changes. This can cause a corrupted current-cloud-backend zip (similar to what @biller-aivy has described)

[brianlenz]

Super helpful, @josefaidt, thanks! I'll let you know once I have a chance to work on this 👍

Great callout, and yes, there is a known instance we are investigating where using the Amplify Hosting feature to connect new frontend apps to existing backends, the backend is undergoing a push when we pushed frontend changes. This can cause a corrupted current-cloud-backend zip (similar to what @biller-aivy has described)

Sounds perfect 🙌