Credentials may not be refreshed on long build.

[ymsrk]

Before opening, please confirm:

• [X] I have checked to see if my question is addressed in the FAQ.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d13pw8k8p2rhkg

Region

ap-northeast-1

Amplify Hosting feature

Build settings

Describe the bug

I am trying to deploy my nextjs app on aws through CodeCommit repository. But the build is failing and giving me below pasted message.

"ExpiredToken: The security token included in the request is expired"

Expected behavior

No exceptions, automatic token refresh if operation takes long.

Reproduction steps

• This occurs after a long build time.
• The probability of this occurring increases when Generating static pages exceeds 6000 pages.

Build Settings

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - amplifyPush --simple
        - rm -rf node_modules
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Additional information

2022-05-13T11:39:57.693Z [INFO]: info  - Generating static pages (0/6189)
2022-05-13T11:40:58.011Z [INFO]: info  - Generating static pages (238/6189)
2022-05-13T11:41:58.353Z [INFO]: info  - Generating static pages (442/6189)
2022-05-13T11:42:58.370Z [INFO]: info  - Generating static pages (751/6189)
2022-05-13T11:43:58.514Z [INFO]: info  - Generating static pages (980/6189)
2022-05-13T11:44:58.553Z [INFO]: info  - Generating static pages (1408/6189)
2022-05-13T11:45:16.698Z [INFO]: info  - Generating static pages (1547/6189)
2022-05-13T11:46:16.947Z [INFO]: info  - Generating static pages (1938/6189)
2022-05-13T11:47:17.183Z [INFO]: info  - Generating static pages (2256/6189)
2022-05-13T11:48:17.386Z [INFO]: info  - Generating static pages (2486/6189)
2022-05-13T11:49:17.494Z [INFO]: info  - Generating static pages (2692/6189)
2022-05-13T11:50:17.641Z [INFO]: info  - Generating static pages (2919/6189)
2022-05-13T11:51:00.133Z [INFO]: info  - Generating static pages (3094/6189)
2022-05-13T11:52:00.264Z [INFO]: info  - Generating static pages (3349/6189)
2022-05-13T11:53:00.329Z [INFO]: info  - Generating static pages (3594/6189)
2022-05-13T11:54:00.581Z [INFO]: info  - Generating static pages (3805/6189)
2022-05-13T11:55:00.636Z [INFO]: info  - Generating static pages (3998/6189)
2022-05-13T11:56:00.690Z [INFO]: info  - Generating static pages (4329/6189)
2022-05-13T11:57:00.806Z [INFO]: info  - Generating static pages (4613/6189)
2022-05-13T11:57:11.404Z [INFO]: info  - Generating static pages (4641/6189)
2022-05-13T11:58:11.519Z [INFO]: info  - Generating static pages (4828/6189)
2022-05-13T11:59:11.553Z [INFO]: info  - Generating static pages (5058/6189)
2022-05-13T12:00:11.829Z [INFO]: info  - Generating static pages (5300/6189)
2022-05-13T12:01:12.448Z [INFO]: info  - Generating static pages (5539/6189)
2022-05-13T12:02:12.681Z [INFO]: info  - Generating static pages (5766/6189)
2022-05-13T12:03:12.821Z [INFO]: info  - Generating static pages (5962/6189)
2022-05-13T12:04:12.891Z [INFO]: info  - Generating static pages (6177/6189)
2022-05-13T12:04:15.921Z [INFO]: info  - Generating static pages (6189/6189)
2022-05-13T12:04:15.934Z [INFO]: info  - Finalizing page optimization...
2022-05-13T12:04:17.330Z [INFO]: 
2022-05-13T12:04:17.396Z [INFO]: Page                                                       Size     First Load JS
                                 ┌ ● / (687 ms)                                             1.21 kB         166 kB
                                 ├   /_app                                                  0 B            69.9 kB

...
2022-05-13T12:04:17.708Z [INFO]: Done in 1503.58s.
2022-05-13T12:04:17.713Z [INFO]: Starting SSR Build...
2022-05-13T12:35:40.334Z [ERROR]: ExpiredToken: The security token included in the request is expired
                                  at Request.extractError (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:686:14)
                                  at Request.transition (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:22:10)
                                  at AcceptorStateMachine.runTo (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:14:12)
                                  at /root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:26:10
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:38:9)
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:688:12)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:686:14)
                                  at Request.transition (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:22:10)
                                  at AcceptorStateMachine.runTo (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:14:12)
                                  at /root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:26:10
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:38:9)
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:688:12)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
                                  at callNextListener (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
                                  at IncomingMessage.onEnd (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/event_listeners.js:335:13)
                                  at IncomingMessage.emit (node:events:402:35)
                                  at IncomingMessage.emit (node:domain:475:12)
                                  at endReadableNT (node:internal/streams/readable:1343:12)
                                  at processTicksAndRejections (node:internal/process/task_queues:83:21) {
                                  code: 'ExpiredToken',
                                  time: 2022-05-13T12:35:40.124Z,
                                  requestId: '826a7bc1-8d34-4081-b410-1711c35d70db',
                                  statusCode: 403,
                                  retryable: true
                                  }
Terminating logging...

[ghost]

Hi @ymsrk 👋🏽 thanks for raising this issue. We are tracking this bug for prioritization. We will update this issue once we have more information

[ghost]

We have added this bug to our backlog for prioritization and the team is investigating a solution for this.

[kayndo]

I am facing the exact issue

[roelvandenbrand]

We are experiencing the same issue. The workaround not always works for us with doing the push locally.

[austinamorusoyardstick]

This is an issue I'm having as well. Is there a way to manually refresh the token?

[austinamorusoyardstick]

Nightwatch test are not finishing due timeout as well https://github.com/aws-amplify/amplify-hosting/issues/3204

[ghost]

Hi @austinamorusoyardstick 👋🏽 the 60 minute timeout is a hard limit at this time. Unfortunately there is no workaround for this but the team is aware of this limitation and are tracking this in our product backlog. We apologize for this inconvenience.

[70ki8suda]

facing same issue

[localsecurity-emily]

facing the same issue

[gauravsapkal1]

@hloriii I am also facing same issue, I am using ISR and have around 25000 pages, my build takes more than hour, and i am getting same issue,

ExpiredToken: The provided token has expired. at Request.extractError (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/services/s3.js:711:35) at Request.callListeners (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/sequential_executor.js:78:10)

@ymsrk your issue resolved ? if yes how ?

[ymsrk]

@gauravsapkal1 Let me tell you how we are getting around this problem. In my project, I limit the number of pages I create to around 25,000 so as not to exceed the 60 minute timeout. I would originally like to create more than 30,000 pages, but I accept the constraint. I hope this problem will be solved in the future.

[gauravsapkal1]

@ymsrk Thanks for comment, I also creating around 25000 pages but now I will limit them upto 10,000.

[Gauravsapkal]

@hloriii if we increase build time in environment variables "_BUILD_TIMEOUT" more than hour will it refresh token after an hour?

[localsecurity-emily]

@hloriii if we increase build time in environment variables "_BUILD_TIMEOUT" more than hour will it refresh token after an hour?

I can say from experience it will not refresh the token.

[Gauravsapkal]

@localsecurity-emily yes I also thinking that it will not work, but I create case at aws they replied this @hloriii can you look into this

[roelvandenbrand]

That works up to one hour until the token expires. A timeout value will cut off the build if you want to make it last like less than 30 minutes max. When set to two hours, the token expiry is simply before that and fails on that. If that issues was not around, it would cut off build time at two hours and then stop.

[Gauravsapkal]

@roelvandenbrand yes I tried its not working by increasing build time, @hloriii can you confirm that increase in _BUILD_TIMEOUT will refresh token or not.

[Jay2113]

Hi everyone, thank you so much for your continued patience and apologies for the delayed response on this thread.

The ExpiredToken: The security token included in the request is expired exception can occur in the following scenarios:

Root cause:

Unfortunately, we do not have a way to increase the token expiration to more than an hour because this limitation is enforced by IAM on the role chaining duration: https://repost.aws/knowledge-center/iam-role-chaining-limit

During the backend build step:

When a backend build is initiated, Amplify Hosting’s build session role assumes your Amplify app’s IAM service role to deploy/update backend resources in your AWS account using temporary security credentials.

Possible workarounds:

• It seems that long backend build duration (> 1hr) can typically occur if your app has a large schema defined in the amplify backend. Our recommendation would be to chunk out or split your changes in the schema. For example, if you were modifying/updating 4-5 relationships at once in your schema, try to update only a couple of relationships at once to see if that reduces the backend build duration.
• Additionally, if your app requires some pre-build and/or post-build steps you can consider using a custom build environment as you can pre-install specific dependencies that usually take longer to install during a build using Amplify's default container and you can speed up this process by creating your own Docker image and reference it during a build.

During the frontend build step:

When a frontend build for a Web_Dynamic (Next.js 11) app (Classic SSR provider) is initiated, Amplify Hosting's build session role assumes your Amplify app's IAM service role to deploy/update backend resources in your AWS account using temporary security credentials.

Possible workarounds:

• Our recommendation would be to reduce the total number of pages you are creating for your Next.js 11 app which thereby could reduce the build duration to < 1hr.
• We strongly recommend that you migrate your Next.js 11 apps to the Amplify Hosting Compute managed SSR provider

---

Note: Updating the _BUILD_TIMEOUT Amplify environment variable to > 60 minutes will not refresh the security token.

[djorgji]

@Jay2113 We are having this is when trying to create new environment is created.

Is there any guidance on scenarios where there isn't an update happening but a create, and the create is timing out?

For us the main issue is the SearchStack taking close to 25 minutes to create, is there to tell it to skip that stack on the first build, or import an existing domain? We could maybe iterate and send a second build out manually, if we were able to somehow keep it from trying to deploy everything the first time around.

Even though that would suck, I would settle for any workaround as we try to figure out and move thing into CDK. We are completely stuck at the moment, our devs are completely blocked.

Thanks in advance.

[pr0g]

I believe our team has hit this issue as well. A developer on our team is attempting to create a new Amplify environment from our existing one, and the creation timed out. I suspect this is happening because we have a relatively complex GraphQL schema based on the advice above. Is there a potential workaround for this? @djorgji were you able to find a solution to this?

@Jay2113 if you have any suggestions I'd be very grateful to hear, thanks very much for your time!