node_modules in generated Lambda for SSR build are missing files

[mfukano]

Before opening, please confirm:

• [X] I have checked to see if my question is addressed in the FAQ.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d3hp54chnllm3g

Region

us-east-1

Amplify Hosting feature

Build settings, Not applicable

Describe the bug

The Lambda@Edge function created after successful deployment of our Next.js app sometimes doesn't download all files from dependencies in node_modules, and the front-end throws HTTPS 503 errors as a result. Is there a way to troubleshoot why this is or ensure complete dependency resolution?

Expected behavior

The preBuild phase should obtain all of the files needed for the generated Lambda function to operate as expected. Something's happening along the way to prevent some dependencies from obtaining all files distributed by these modules at install time during the build phase.

Reproduction steps

Our actual top-level dependencies are fewer than our environment dependencies so I'll list them below:

{
  "name": "asdf",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@emotion/react": "^11.9.0",
    "@emotion/styled": "^11.8.1",
    "@mui/icons-material": "^5.8.0",
    "@mui/lab": "^5.0.0-alpha.85",
    "@mui/material": "^5.6.4",
    "@next/bundle-analyzer": "^12.2.5",
    "@stripe/react-stripe-js": "^1.8.1",
    "@stripe/stripe-js": "^1.31.0",
    "aws-amplify": "^4.3.35",
    "chart.js": "^3.7.1",
    "gl-matrix": "^3.4.3",
    "imagemin-gifsicle": "^7.0.0",
    "imagemin-optipng": "^8.0.0",
    "imagemin-svgo": "^9.0.0",
    "lqip-loader": "^2.2.1",
    "mapbox-gl": "^1.13.2",
    "next": "^12.3.1",
    "next-optimized-images": "^2.6.2",
    "prop-types": "^15.8.1",
    "react": "^18.2.0",
    "react-chartjs-2": "^4.1.0",
    "react-dom": "^18.2.0",
    "react-ga4": "^1.4.1",
    "react-markdown": "^6.0.0",
    "react-router": "^6.3.0",
    "stripe": "^9.8.0",
    "svg-sprite-loader": "^6.0.11",
    "typeface-roboto": "^1.1.13",
    "typescript": "^4.6.4",
    "use-debounce": "^8.0.2",
    "use-deep-compare-effect": "^1.8.1",
    "web-vitals": "^2.1.4",
    "webp-loader": "^0.6.0"
  },
  "overrides": {
    "**/@emotion/cache/stylis": "4.0.5"
  },
  "devDependencies": {
    "@babel/eslint-parser": "^7.19.1",
    "@next/eslint-plugin-next": "^12.3.0",
    "@types/node": "18.0.6",
    "@types/react": "^17.0.2",
    "@types/react-dom": "18.0.6",
    "cypress": "10.3.1",
    "eslint": "^8.23.1",
    "eslint-config-next": "^12.3.0",
    "eslint-plugin-cypress": "^2.12.1",
    "eslint-plugin-react": "^7.31.8",
    "start-server-and-test": "1.14.0"
  },
  "scripts": {
    "dev": "next dev",
    "start": "next start",
    "build": "next build",
    "lint": "next lint --quiet",
    "cypress": "cypress open",
    "cypress:headless": "cypress run",
    "e2e": "start-server-and-test dev http://localhost:3000 cypress",
    "e2e:headless": "start-server-and-test dev http://localhost:3000 cypress:headless"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

a. We're running Next.js 12 but I'm not sure how much a factor that is here. b. Our Amplify integration doesn't rely on @aws-amplify/interactions at all, at least not explicitly.

1. Run full build & deployment flow on Amplify.
2. Assuming the deployment with the above packages has successfully completed, routing to the site should be enough to trigger this repro; it occurs immediately and doesn't attempt to load anything aside from the main HTML entrypoint when the 503 occurs.
3. Navigate to Lambda function generated by Next/Amplify build and deployment.
4. Determine problematic module by examining CloudWatch logs.
5. Obtain .zip from Lambda export and check problematic module to see if files are missing.

Build Settings

version: 1
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - ls -a
        - rm -rf node_modules
        - rm -rf .next
        - node -v
        - npm -v
        - nvm use --lts
        - node -v
        - npm -v
        - npm ci
    build:
      commands:
        - nvm use --lts
        - node -v
        - npm -v
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Additional information

The full problem

Amplify deploys both our back-end and front-end successfully, but upon attempting to load our development environment's page, an HTTP 503 response is thrown with relatively generic error text:

503 ERROR
The request could not be satisfied.
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Inspecting CloudWatch exposes an error of type MODULE_NOT_FOUND on fflate, a dependency of @aws-amplify/interactions as seen by the below stacktrace:

ERROR	Invoke Error 	
{
    "errorType": "Error",
    "errorMessage": "Cannot find module '/var/task/node_modules/fflate/lib/node.cjs'",
    "code": "MODULE_NOT_FOUND",
    "path": "/var/task/node_modules/fflate/package.json",
    "stack": [
        "Error: Cannot find module '/var/task/node_modules/fflate/lib/node.cjs'",
        "    at createEsmNotFoundErr (internal/modules/cjs/loader.js:842:15)",
        "    at finalizeEsmResolution (internal/modules/cjs/loader.js:835:15)",
        "    at resolveExports (internal/modules/cjs/loader.js:424:14)",
        "    at Function.Module._findPath (internal/modules/cjs/loader.js:464:31)",
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:802:27)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:667:27)",
        "    at Module.require (internal/modules/cjs/loader.js:887:19)",
        "    at require (internal/modules/cjs/helpers.js:74:18)",
        "    at Object.<anonymous> (/var/task/node_modules/@aws-amplify/interactions/lib/Providers/AWSLexProviderHelper/utils.js:51:16)",
        "    at Module._compile (internal/modules/cjs/loader.js:999:30)"
    ]
}

After exporting the Lambda as a zip and unpacking node_modules, I found that the diff between this module and the one I have locally installed (on the same project with the same file-set as the git commit that triggered CI/CD) is ~3 JS files needed to correctly execute this call. I went through several modules to correct file references and was able to restore behavior to the development environment but this can't happen on every deploy.

[multivoltage]

Maybe in you locale pc "nvm use --lts" switch to different version?

[mfukano]

Hey @hloriana! Have you and the team been able to take a look into this and/or reproduce the problem? I'm curious if this is a problem in our specific build flow or if the way that the node_modules are deployed to Lambda reduces package size by design, thus causing this specific issue for us. Thanks!

[mfukano]

I figured out what was going on with this — @multivoltage was correct in that there was a slight mismatch between the Node 16 LTS version being used on the Amplify build step and my own Node 16 LTS version installed, which caused some dependencies to partially install / fail silently. After resolving this, there seemingly haven't been further issues with the build or deployment stages regarding partial dependencies.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.