serverless-next.js icon indicating copy to clipboard operation
serverless-next.js copied to clipboard

New IAM roles created every deployment -- causes 503's when 1000/account limit reached (LimitExceeded)

Open patricktyndall opened this issue 3 years ago • 5 comments
trafficstars

Issue Summary

When my app deploys, 2 new IAM roles are created every deployment. I've committed all deployment diff, redeployed, and this still happens.

My use cases is automated daily deploys from CI.

  • 2 new roles created with each deployment (once a day)
  • If committing output is necessary, having to create automated commits each day

When you try to deploy when you're at the limit, it fails with message "LimitExceeded: Cannot exceed quota for RolesPerAccount: 1000". Also, my app started serving cloud front error pages with the same message. So this breaks the site.

So two separate things here:

  • App creating infinite IAM roles, leading to build failures
  • Build failures breaking existing infra

Actual behavior

  • App creating 2 IAM roles every deployment, leading to build failures when the limit is reached
    • This is the more important issue
  • Build failures breaking existing infra

Expected behavior

Deployments do not create IAM roles, are able to use existing ones.

Steps to reproduce

Get to 1k IAM roles in your AWS account, then try to deploy using this component

Screenshots/Code/Configuration/Logs

  error:
  LimitExceeded: Cannot exceed quota for RolesPerAccount: 1000
    at Request.extractError (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/root/.serverless/components/registry/npm/@sls-next/[email protected]/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'LimitExceeded',
  time: 2021-12-24T16:45:53.066Z,
  requestId: '6aab5cc3-3bae-405b-bc75-f98214f33664',
  statusCode: 409,
  retryable: false,
  retryDelay: 22.3930525852321
}

Versions

  • OS/Environment: Mac/Linux
  • @sls-next/serverless-component version: @sls-next/[email protected]
  • Next.js version: ^10.2.4-canary.2

patricktyndall avatar Dec 27 '21 16:12 patricktyndall

Also reported here - https://github.com/serverless-nextjs/serverless-next.js/issues/1698

jchou1271 avatar Jan 03 '22 16:01 jchou1271

Same issue here (@sls-next/[email protected])

cweekly avatar Feb 04 '22 20:02 cweekly

I am no longer having this issue on @sls-next/[email protected].

The fix was either the version bump, or committing of build artifacts (apologies, it's been a while).

However I AM still having this issue with Lambdas being created on each deploy. This leads to a similar failure scenario where my AWS account's lambda storage is full. Anyone seeing that as well?

patricktyndall avatar Feb 23 '23 16:02 patricktyndall

I moved on from the project where I saw this per my comment 2022-02-04 above, but in case it helps, there may be a solution / mitigation: https://github.com/serverless-nextjs/serverless-next.js/issues/1698#issuecomment-1057936229

cweekly avatar Apr 09 '23 20:04 cweekly