Next.js development high memory usage

[timneutkens]

Before posting a comment on this issue please read this entire post.

Previous work

The past few weeks we've been investigating / optimizing various memory usage issues. Specifically geared towards production memory usage. In investigating these we were able to find there was one memory leak in Node.js itself when using fetch() in Node.js versions before 18.17.0 (you'll want to use 18.17.1 for security patches though).

Most of the reports related to memory usage turned out to be reports of "it's higher than the previous version" rather than a memory leak. This was expected because in order to run App Router and Pages Router at the same time with different React versions two separate processes were needed. This has been resolved by reducing the amount of processes to two, one for Routing and App Router rendering, and one for Pages Router rendering. So far we haven't received new reports since the latest release.

In some issues there were reports related to Image Optimization in production, however no reproduction was provided so it could not be investigated adequately, if you have a reproduction for that please refer to this issue: #54482

New

With the memory usage in production resolved we've started investigating reports of development memory usage spikes. Unfortunately these reports suffer from the same problem as the production memory usage issues people raised before, they're full of comments saying same issue or posting screenshots of monitoring tools saying "Look, same issue".

Unfortunately, as you can image, these replies are not enough to investigate / narrow down what causes the memory usage, for example in multiple cases that we did get a reproduction and could investigate the reason for the high memory usage was:

• A bug in the application code, causing infinite looping components
• Accidental import of ~11.000 modules through icon libraries -- Yes, many icon libraries ship massive amounts of re-exports and those have to be bundled before it can be tree-shaken. We've been working on an automated way to split these up in #54572 that should help a bit to reduce the size (and compilation speed too).
• Webpack customization, e.g. adding external libraries that change webpack settings, for example to write additional sourcemaps

So far I've been able to make one small change to webpack's memory caching to make it garbage collect a bit more aggressively in #54397. I'm not expecting that change to have a big impact on the reported issues though.

We'd like to investigate these reports further, however we're unable to narrow these down if there is no code to run to collect heap snapshots and profiles for, hence this issue. If you are able to please provide runnable code of what you're experiencing.

Comments that don't include runnable code will be automatically hidden in order to keep this issue productive. This includes comments that only have a screenshot and applications that can't run.

I'm going to combine the other reports into this issue as separate comments.

I've made sure that we have 2-3 engineers on our team available to investigate when we get runnable reproductions to investigate.

Thanks in advance!

_NEXT-1569

[tomelliot]

Thanks for the attention on this topic @timneutkens. I've been experiencing this issue pretty consistently in dev. I'm not sure I'll be able to create a reproduction for it - it seems more to do with the hot-reload operation than with any specific code I'm writing. Is there a way for me to collect some relevant debug info locally until I run into the issue again, and share that instead?

[tomelliot]

I understand your frustration with all the "same" comments, but I'm sorry to say I'm doing my best to be helpful here @timneutkens. I'm not an experienced enough developer to get to the bottom of this myself, and I was just offering the best I could to try and support by collecting data locally.

I'll leave it to you and the pros.

[AhmedChabayta]

i was building an app using the app directory, and i kept getting server out of memory in console, and some sort of input error in the browser, the server crashes and thats that, no errors beyond the ones mentioned, furthermore, the app was slow in loading, i created another app using pages this time, kept everything exactly the same just made it the proper pages format, and all the issues have gone. no more server out of memory, nor slow loading times. Next.js is and most probably will remain my personal favorite react framework, but this shit is ridiculous. maybe we get an explanation at least to why this is happening? i spent f** hours trying to find out what i did wrong, only to realize the problem isnt from my code.

i updated Nextjs to the latest version, updated node, used different versions of node via nvm, removed my code block by block to see if the issue is coming from some broken piece of code, same shit. the only solution was to move on from the app directory.

[timneutkens]

maybe we get an explanation at least to why this is happening?

It's impossible to say what you're running into without code being shared. It could be you import a library in both server components and client components which means it's bundled twice compared to pages and such. Given that you have a reproduction please share it (doesn't have to be public, you can give me / my team access). As said before we'd be happy to take a look.

[AhmedChabayta]

https://github.com/AhmedChabayta/social.me

if you can tell me there's something wrong with my code, id forever appreciate it for helping me move on from this issue.

thank you in any case.

[timneutkens]

@AhmedChabayta It looks like I'll need a Sanity key in order to run the project, can you provide that?

Also do you have a branch with the change to use pages instead? Just so that I can compare them.

[AhmedChabayta]

@timneutkens is it possible i send the keys in private? and no, i made a whole new repo locally to test on pages. i sincerely appreciate your help. thank u

[timneutkens]

@AhmedChabayta feel free to send them on https://twitter.com/timneutkens.

i made a whole new repo locally to test on pages. i sincerely appreciate your help. thank u

Can you share that repository too? Want to make sure I can compare them.

[AhmedChabayta]

@timneutkens i should add this, i tried removing the sidebar all together, in case the icons might be the ones causing the issue, but the server still kept crashing anyway. i just sent the links to the 2 repos, and the whole bunch of keys i used in both apps.

[timneutkens]

Thanks! Added both to our internal copy of this issue so that your keys don't leak. We'll take a look!

[noetix]

Its interesting that the downvote emojis on the bullying post have been removed.

All the dude did was ask how to collect data in a way that would be helpful. If collecting our own data is out of the question then that should have been the answer.

Poor form for someone representing Vercel.

[weyert]

Do you expect us to verify the leakage in canary first?

[Thinkscape]

Upgrading from [email protected] to [email protected] didn't solve the memleak, however it broke the server restart. In .12 I'd get the (too often) memory running out warning, then the server would restart and a refresh in the browser would work (system user mem usage would dip by a few GB after restart)

[timneutkens]

@noetix Disagree that this had anything to do with bullying. All I did was explain that my posts are continuously being ignored, including in this issue. Then I explained what to do, which is to provide a runnable application. There is nothing we can do without a runnable example, which was already shared in the initial issue, I even made it bold to highlight that further.

Happy to explain it again, the reason we can't do anything without runnable code is that in order to narrow down the memory usage we need to change the Next.js core code in the application, for example to disable client components compilation and such in order to narrow down where the memory usage comes from. There is no way to do that based on screenshots / messages / information you can provide as it would require countless hours of your time and our time (think 2 weeks full time at least) in order to investigate this.

The emoji reactions not being shown for off-topic marked posts is a bug in GitHub. As mentioned in the initial issue any posts that don't include a reproduction will be automatically hidden.

Since you didn't like the earlier explanation I'll just remove it, don't feel strongly about keeping the comment. Definitely wasn't bullying, you were reading into that. Bullying would be the threats I've received recently from anonymous developers on Twitter that they'll come visit my house soon...

@weyert we haven't made changes to development memory usage besides the PR linked in the issue so really all I need is a reproduction, luckily @AhmedChabayta posted one, hopefully that is enough, fingers crossed.

@Thinkscape please open a separate issue, that bug would be separate from this issue 👍

[timneutkens]

@Thinkscape I've opened a PR for what you were reporting, managed to reproduce it by artificially triggering a reboot. Here's the PR: https://github.com/vercel/next.js/pull/54760

Since you mentioned that you run into the watcher behavior often please provide a reproduction of that too so that we can investigate it.

[limeburst]

I've posted a reproduction here: https://github.com/limeburst/vercel-nextjs-54708

Start the development server, navigate from

• http://localhost:3000/1
• http://localhost:3000/2
• ...
• http://localhost:3000/20

And watch the memory usage grow, until the server restarts.

[DiegoCantu]

Will this be related to the following? Detected multiple renderers concurrently rendering the same context provider. https://github.com/vercel/next.js/discussions/37424

maybe it goes hand in hand ..

[AhmedChabayta]

i downgraded to 13.4.11 and the server stopped restarting, i could be wrong, but I'm going to assume at least I'm not doing anything to crash it.

[marioolofo]

I started some projects with version 13.4.7 and never had any crash or OOM, but with the latest versions I have a lot of error 500 and OOM. Unfortunatelly I don't know how to make a simple use case to reproduce because the problem occour after some time editting any page. Anyways, the version 13.4.7 is different in some way, I though it was valid to give a heads up about it.

[rajat1saxena]

You can use this code to see the issue. The server is getting aborted silently without any errors.

https://github.com/codelitdev/courselit/tree/tailwindcss-2

Logs

rajat@rajat-laptop:~/projects/courselit$ yarn dev
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- ready started server on [::]:3000, url: http://localhost:3000
- event compiled client and server successfully in 545 ms (18 modules)
- wait compiling...
- event compiled client and server successfully in 263 ms (18 modules)
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- wait compiling /404 (client and server)...
- wait compiling / (client and server)...
rajat@rajat-laptop:~/projects/courselit$

[ab70]

I can see there are too many next-router-worker and that takes more than 3GB++ ram,

this is the log while it stopped the server. `- warn The server is running out of memory, restarting to free up memory.

niam_next_reqrite on  v2 [!?] via  v19.9.0 took 3h19m26s ` by this time my my 16GB ram wasn't enough for this dev. I was using latest stable release then to solve i switch to canary version, but it didn't help. Currently, can't provide any reproductions, surely will drop one soon.

using these dependencies in package.json:

"dependencies": {
    "@ant-design/icons": "^5.1.4",
    "@ant-design/plots": "^1.2.5",
    "@tanstack/react-query": "^4.29.13",
    "@tanstack/react-query-devtools": "^4.29.13",
    "antd": "^5.6.1",
    "axios": "^1.4.0",
    "csv-parser": "^3.0.0",
    "dayjs": "^1.11.9",
    "dotenv": "^16.1.4",
    "next": "^13.4.20-canary.10",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-icons": "^4.10.1",
    "react-router-dom": "^6.12.0",
    "sharp": "^0.32.1",
    "xlsx": "^0.18.5"
  }

And node version: v19.9.0

[stigmartins1]

I'm getting 'server out of memory' after a while by letting the server run and writing/saving code that calls the following functions a few times.

import { google } from "googleapis";

export async function authSheets() {
  //Function for authentication object
  const auth = new google.auth.GoogleAuth({
    keyFile: "./auth/auth-sa-sptk.json",
    scopes: ["https://www.googleapis.com/auth/spreadsheets"],
  });

  //Create client instance for auth
  const authClient = await auth.getClient();

  //Instance of the Sheets API
  const sheets = google.sheets({ version: "v4", auth: authClient });

  return {
    auth,
    authClient,
    sheets,
  };
}

import { authSheets } from "./authSheets";

export async function clearSheetContents(sheetName) {
  console.log("sheet =", sheetName);
  const SHEET_ID = "123";
  const sheetId = SHEET_ID;
  const { sheets } = await authSheets();

  try {
    const result = await sheets.spreadsheets.values.clear({
      spreadsheetId: sheetId,
      range: sheetName,
    });
    console.log("result.data =", result.data);
  } catch (err) {
    // TODO (developer) - Handle exception
    throw err;
  }
}

import { authSheets } from "./authSheets";

// https://developers.google.com/sheets/api/guides/values
export async function setSheetValues(sheetName, input) {
  const SHEET_ID = "123";
  const sheetId = SHEET_ID;
  const values = [input];
  const resource = { values };
  // Updates require a valid ValueInputOption parameter
  const valueInputOption = "RAW"; // The input is not parsed and is inserted as a string.
  const { sheets } = await authSheets();

  try {
    const result = await sheets.spreadsheets.values.append({
      spreadsheetId: sheetId,
      range: sheetName,
      valueInputOption: valueInputOption,
      resource,
    });
    console.log("result.data =", result.data);
  } catch (err) {
    // TODO (developer) - Handle exception
    throw err;
  }
}

{
  "name": "test",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "autoprefixer": "10.4.15",
    "axios": "^1.5.0",
    "encoding": "^0.1.13",
    "eslint": "8.48.0",
    "eslint-config-next": "13.4.19",
    "googleapis": "^126.0.1",
    "next": "13.4.19",
    "postcss": "8.4.29",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "tailwindcss": "3.3.3"
  }
}

Node v. v18.17.1

[na0495]

👋 Hello, I'm encountering a similar issue as described in #54708. I'd like to share my use case and the error logs to provide more context. Environment

OS: Ubuntu 22.04.3 LTS
Node.js version: v20.5.1
Next.js version: ^13.4.19

Use Case

I'm working on a [describe your project or feature briefly]. When I first run the development server, everything works as expected. However, after 5-15 minutes of development and applying code changes, the app runs out of memory. Steps to Reproduce

Start the development server with yarn dev (next dev -p 3000).
Apply  code changes and save for 5-15 minutes.
Observe the server running out of memory.

Error Logs

Here are the error logs I encounter:

- warn The server is running out of memory, restarting to free up memory.
- wait compiling...
TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeRequest (/home/ubuntu/workspace/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
    at async invokeRender (/home/ubuntu/workspace/app/node_modules/next/dist/server/lib/router-server.js:254:29)
    at async handleRequest (/home/ubuntu/workspace/app/node_modules/next/dist/server/lib/router-server.js:447:24)
    at async requestHandler (/home/ubuntu/workspace/app/node_modules/next/dist/server/lib/router-server.js:464:13)
    at async Server.<anonymous> (/home/ubuntu/workspace/app/node_modules/next/dist/server/lib/start-server.js:117:13) {
  cause: Error: connect ECONNREFUSED 127.0.0.1:46251
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '127.0.0.1',
    port: 46251
  }
}
- wait compiling /_error (client and server)...

Additional Context

I've tried other node.js version from 18 to 20, the issue seems to stay.
This issue is blocking my development workflow.

Would appreciate any guidance or fixes for this issue. Thank you!

[andy-leezard]

I've posted a reproduction here: https://github.com/limeburst/vercel-nextjs-54708

Start the development server, navigate from

• http://localhost:3000/1
• http://localhost:3000/2
• ...
• http://localhost:3000/20

And watch the memory usage grow, until the server restarts.

I confirm. Navigate from http://localhost:3000/1 to http://localhost:3000/20.

- wait compiling /20/page (client and server)...
- warn ./node_modules/node-fetch/lib/index.js
Module not found: Can't resolve 'encoding' in 'D:\dev\next\vercel-nextjs-54708\node_modules\node-fetch\lib'        

Import trace for requested module:
./node_modules/node-fetch/lib/index.js
./node_modules/gaxios/build/src/gaxios.js
./node_modules/gaxios/build/src/index.js
./node_modules/googleapis-common/build/src/index.js
./node_modules/googleapis/build/src/index.js
./lib/util.ts
./app/17/page.tsx
- warn The server is running out of memory, restarting to free up memory.
TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeRequest (D:\dev\next\vercel-nextjs-54708\node_modules\next\dist\server\lib\server-ipc\invoke-request.js:17:12)
    at async invokeRender (D:\dev\next\vercel-nextjs-54708\node_modules\next\dist\server\lib\router-server.js:254:29)
    at async handleRequest (D:\dev\next\vercel-nextjs-54708\node_modules\next\dist\server\lib\router-server.js:447:24)
    at async requestHandler (D:\dev\next\vercel-nextjs-54708\node_modules\next\dist\server\lib\router-server.js:464:13)
    at async Server.<anonymous> (D:\dev\next\vercel-nextjs-54708\node_modules\next\dist\server\lib\start-server.js:117:13) {
  cause: Error: connect ECONNREFUSED ::1:53698
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -4078,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 53698
  }
}

[DiegoCantu]

I installed 18.17.1 and the issue still happen with more frequency than my previous version but now I don't' remember wich was ...

[nick-myers-dt]

My comment wasn't relevant, so I have deleted the content, thank you @kachkaev

[kachkaev]

I'm running Node v18.10.0

Please see the first comment 🙏

In investigating these we were able to find there was one memory leak in Node.js itself when using fetch() in Node.js versions before 18.17.0 (you'll want to use 18.17.1 for security patches though).

[na0495]

@kachkaev the message isn't really clear, do you suggest using a higher version then 18.17.1 ?

[kachkaev]

@na0495 my reply was related to a comment by @nick-myers-dt, which he has edited afterwards. My suggestion here matches what Tim says in the first comment: update Node from 18.× (e.g. 18.10.0) to 18.17.0 or 18.17.1 before reporting memory leaks in dev.

[feedthejim]

@stigmartins1 I investigated your scenario, and I landed a fix that should slightly improve the situation in that specific scenario.

Basically what happens is that Next.js will compile the googleapis module as part of your code. Unfortunately, that module is huge and ends up adding upward to 200MB when loaded in memory. I added a tweak that should alleviate that cost. Please let me know if it improved.