NextJs compiling extremely slow

[stevensturkop]

trafficstars

What version of Next.js are you using?

11.1.2

What version of Node.js are you using?

14.18.0

What browser are you using?

Chrome

What operating system are you using?

Windows 10

How are you deploying your application?

AWS ECS

Describe the Bug

I've been using NextJs for years and recently it has been very hard to work with because very slow in development. After npm run dev, I go to localhost:3000. From there the page can take up to 60 seconds to display. Then when the first page finally displays, each code change fast refresh or page transition SSR (compilation) takes between 15-20 seconds, sometimes more than 30 seconds, and sometimes it even doesn't work so I have to refresh the page.

• I read that on windows, windows defender could be blocking so I did this, which didn't help: https://github.com/vercel/next.js/issues/12797#issuecomment-660225689
• I use webpack 5
• I read tailwind could be causing this kind of issue, but I don't use it.

Expected Behavior

Page load + compilation should be faster.

To Reproduce

Unfortunately, I cannot send a reproducible UI because the project I work on is pretty big and the UI is under NDA.

Maybe without a reproducible UI, you guys have thoughts about how to improve/fix the compilation time on windows. Maybe some of you have faced the same problem and fixed it somehow. I'm all ears.

My package.json

{ "name": "myname", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start -p 3002" }, "dependencies": { "@react-google-maps/api": "^2.0.2", "@sentry/browser": "^6.7.2", "@sentry/react": "^6.7.2", "@sentry/tracing": "^6.7.2", "@sentry/webpack-plugin": "^1.15.1", "@stripe/react-stripe-js": "^1.4.0", "@stripe/stripe-js": "^1.13.1", "accept-language-parser": "^1.5.0", "aws-sdk": "^2.802.0", "base-64": "^1.0.0", "cors": "^2.8.5", "dotenv": "^8.2.0", "hls.js": "^0.14.16", "iban": "0.0.14", "js-sha256": "^0.9.0", "libphonenumber-js": "^1.9.11", "localized-countries": "^2.0.0", "lodash": "^4.17.21", "moment": "^2.29.1", "next": "^11.1.2", "next-compose-plugins": "^2.2.1", "next-fonts": "^1.5.1", "next-images": "^1.8.1", "next-seo": "^4.17.0", "nookies": "^2.5.0", "platform": "^1.3.6", "qrcode": "^1.4.4", "randomstring": "^1.1.5", "react": "^17.0.2", "react-cropper": "^2.1.4", "react-datepicker": "^3.3.0", "react-device-detect": "^1.15.0", "react-dom": "^17.0.2", "react-dotdotdot": "^1.3.1", "react-ga": "^3.3.0", "react-geocode": "^0.2.2", "react-google-maps": "^9.4.5", "react-lines-ellipsis": "^0.14.1", "react-loading-skeleton": "^2.2.0", "react-query": "^3.13.2", "react-textarea-autosize": "^8.3.2", "sass": "^1.29.0", "stripe": "^8.137.0", "uuid": "^8.3.1", "video.js": "^7.11.0", "videojs-contrib-dash": "^4.0.0" }, "devDependencies": { "@svgr/webpack": "^5.5.0" } }

[Philreact]

I've had the same problem for the last 8 months. It seems many people are having the same problem. In my opinion this should be a priority to get fixed. I have tried almost everything: deactivating windows defender, using yarn instead of npm, but nothing has worked.

sometimes it takes 1 min to change pages in development. In production it's super fast.

I can't publish my code as well since it's copyrighted.

here are my dependencies

{
  "name": "",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "next": "next",
    "dev": "nodemon server/index.js",
    "build": "next build",
    "start": "node server/index.js",
    "analyze": "cross-env ANALYZE=true next build",
    "analyze:server": "cross-env BUNDLE_ANALYZE=server next build",
    "analyze:browser": "cross-env BUNDLE_ANALYZE=browser next build"
  },
  "dependencies": {
    "@date-io/date-fns": "^1.3.13",
    "@dnd-kit/core": "^3.1.1",
    "@dnd-kit/sortable": "^4.0.0",
    "@material-ui/core": "^4.12.3",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.60",
    "@material-ui/pickers": "^3.3.10",
    "@next/bundle-analyzer": "^11.0.1",
    "@sendgrid/mail": "^7.4.0",
    "@stripe/stripe-js": "^1.12.1",
    "@tippyjs/react": "^4.2.5",
    "aws-sdk": "^2.809.0",
    "axios": "^0.21.0",
    "bcryptjs": "^2.4.3",
    "bootstrap": "^4.4.1",
    "chroma-js": "^2.1.1",
    "config": "^3.3.2",
    "cookie": "^0.4.1",
    "cookie-parser": "^1.4.5",
    "cookie-session": "^1.4.0",
    "cors": "^2.8.5",
    "cross-env": "^7.0.3",
    "d3-ease": "^2.0.0",
    "date-fns": "^2.21.1",
    "express": "^4.17.1",
    "express-session": "^1.17.1",
    "express-validator": "^6.6.1",
    "geoip-lite": "^1.4.2",
    "intersection-observer": "^0.12.0",
    "js-cookie": "^2.2.1",
    "jsonwebtoken": "^8.5.1",
    "moment": "^2.29.1",
    "mongoose": "^5.10.11",
    "next": "10.2.3",
    "next-absolute-url": "^1.2.2",
    "node-fetch": "^2.6.1",
    "nprogress": "^0.2.0",
    "passport": "^0.4.1",
    "passport-google-oauth20": "^2.0.0",
    "password-validator": "^5.1.1",
    "pdfjs-dist": "^2.7.570",
    "prismjs": "^1.23.0",
    "quill": "^1.3.7",
    "randomcolor": "^0.6.2",
    "react": "16.14.0",
    "react-beautiful-dnd": "^11.0.5",
    "react-bootstrap": "^1.4.0",
    "react-contenteditable": "^3.3.5",
    "react-dom": "16.14.0",
    "react-dropzone": "^11.2.3",
    "react-quill": "^1.3.5",
    "react-select": "^3.1.1",
    "react-spring": "^8.0.27",
    "react-toastify": "^6.1.0",
    "react-zoom-pan-pinch": "^1.6.1",
    "request-ip": "^2.1.3",
    "roughjs": "^4.3.1",
    "sass": "^1.27.0",
    "socket.io": "^3.1.2",
    "socket.io-client": "^3.1.2",
    "string-strip-html": "^4.3.5",
    "stripe": "^8.137.0",
    "validator": "^13.5.2",
    "web-push": "^3.4.4"
  },
  "devDependencies": {
    "eslint": "^7.16.0",
    "eslint-plugin-react": "^7.21.5"
  }
}

[balazsorban44]

Is this the same for a fresh create next app as well? it could potentially point out if the problem is with one of the dependencies maybe or something special with your machine and next.

https://nextjs.org/docs/api-reference/create-next-app

[Philreact]

Is this the same for a fresh create next app as well? it could potentially point out if the problem is with one of the dependencies maybe or something special with your machine and next.

https://nextjs.org/docs/api-reference/create-next-app

In my case I have a custom express server. It's when I started using getServerSideProps that it's been slow only in development. I have 2 other people also coding in this project and in their machines the same problem persists.

The weird thing is that in production it's lightning speed.

[balazsorban44]

Production is very different from a dev server. If your teammates experience the same problem, it's more likely that the problem is in your setup. Maybe a slow database call in the test environment where the DB has less resources?

Without a reproduction though, I can only guess. you could try to create a minimal reproduction that does not use code directly from work but still produces the same effect. you might even find the reason for the issue along the way.

[stevensturkop]

@balazsorban44 We are using DynamoDB hosted on an ARM AWS ECS large which is pretty fast. So I believe the database is not the problem at all (especially as we query only 50 elements at a time, so it's pretty light). A minimal reproduction wouldn't showcase the problem.

We have around 500 components in the project, so all things considered I'm wondering whether NextJs has some limitations where it cannot handle compiling in development big files/pages?

[stevensturkop]

I just analyzed the bundle, despite the very slow first page init after npm run dev (around 1minute) the targetted page Size is 120kB and the page first load JS is 1.2MB.

[timneutkens]

Hey Steven, could you send me a message on https://twitter.com/timneutkens, then we can investigate it without code access.

[timneutkens]

@PhillipLangMartinez same for you

[kartikrao]

Our app www.drive.com.au is the largest automotive site in Australia.

Creating a production build with next build takes > 20 minutes on a machine with 6 cores and 32 GB of RAM. Our builds have slowed significantly since updating to Next 11.x

@timneutkens - Happy to provide access to config and code, if its of wider benefit to the community, we are also happy to pitch in if helpful.

Issue template

What version of Next.js are you using?

11.1.2

What version of Node.js are you using?

14.18.0

What browser are you using?

Chrome

What operating system are you using?

Mac OS 11.6 (Big Sur)

How are you deploying your application?

AWS ECS

Describe the Bug

We use getServerSideProps for > 90% of pages.

We've tried:

• Enabling and disabling build caching (.next folder)
• Changing NodeJS version (14.16)
• Upgrading/Downgrading dependencies - babel, postcss, tailwind etc.

[timneutkens]

Could you reach out to me on twitter as well, thanks!

[kartikrao]

Could you reach out to me on twitter as well, thanks!

@timneutkens - My apologies, the only social network I'm on is LinkedIn, is there an alternative way to get in touch ?

My email is redacted

[timneutkens]

Sent you a short email with what to run 👍

[billymcintosh]

I admire you helping one to one, but shouldn't the potential solutions be posted here to help everyone that has this issue?

[f16z]

so much pain in development.. still hunting for solutions

[timneutkens]

Experiencing this issue?

To provide a trace which only includes metadata of the application:

• npm install next@canary (or yarn add next@canary)
• Run development like normal and reproduce the issue you're experiencing
• Stop Next.js (generally ctrl + c)
• Send the file .next/trace file here or send it to https://twitter.com/timneutkens if you don't want to share it publicly. You can use https://gist.github.com to create a private url for the file. - If you have a custom .babelrc please provide it - If you have a custom next.config.js please provide it - if you are using TailwindCSS please provide tailwind.config.js

⚠️ The metadata in .next/trace includes full file paths to all JS/CSS. It does not include the file contents. ⚠️ The main thing we need to help investigate your application is .next/trace, package.json and such are not helpful in investigating the issue.

Reply to earlier messages

but shouldn't the potential solutions be posted here to help everyone that has this issue?

So far all I've done is help investigate the slowdowns for the people that reached out. There's no specific solution as it highly depends on the app and I'm going to share the findings. . I've been working on a way that we can investigate these slowdowns without needing the application code itself but it requires a bunch of steps that up till a few days ago were quite involved.

Here's a list of what I've found so far:

• @stevensturkop hasn't provided a trace yet
• @PhillipLangMartinez
  • Fast Refresh is behaving as expected, no particular slowdown, will benefit from the work we're doing for SWC and webpack optimizations though
  • The main problem they're experiencing is about on-demand-entries. Next.js compiles the JS/CSS/etc for pages on-demand when you request them, so if you request http://localhost:3000/about Next.js will at that point start compiling the JS/CSS for pages/about.js, this is intentional to ensure that you can have thousands of pages in a Next.js app and it would not slow down your app, otherwise you'd pay this compilation penalty at bootup
    • This will also benefit from the work we're doing integrating SWC with full backwards compat. I've had Phillip try the new flag and it's resulted in at least 50% faster initial compilation and ~60% faster compiling of pages. Note that this highly depends on if you have a lot of node_modules or a lot of application code, the application code will be compiled faster but node_modules are not passed through babel currently so they are not affected by the work we're doing to add SWC at this point in time.
  • In order to investigate this one further I'm adding timings for filesystem reads to find if that's what is causing issues with a large amount of JS as they have a bunch of large node_modules like react-bootstrap included
• @kartikrao
  • 90% of the time spent compiling is related to TailwindCSS, it's slowing down each CSS file by spending about 2 minutes and 30 seconds, it seems that the application is not using TailwindCSS JIT, so I'd recommend enabling that for all users that are using TailwindCSS: https://tailwindcss.com/docs/just-in-time-mode
• @f16z
  • Machine fs.readFile seems to be slow, it takes over 100ms per file, we expanded the tracing to include timing for individual files
  • Initial compile + compiling pages/index.js reads and compiles about 6867 files. What I've found is that 5557 of those are coming from @material-ui/icons/esm, assuming that the app is not using 5.5K icons I'd recommend changing how they are imported so that not all files have to be compiled. Based on debugging this trace we're adding a separate chart that will allow us to visualize the dependency graph based on a trace file
• @billymcintosh can you provide a trace based on the above instructions

[f16z]

I have issues installing next@canary until I ran npm config set legacy-peer-deps true

[AdamDiament]

• Send the file .next/trace file here or send it to https://twitter.com/timneutkens if you don't want to share it publicly

@timneutkens Thanks for looking into this. I am about to send you my files privately on twitter. I love working with next but this is a major pain point, so I really appreciate that you are prioritising it. During the time the project was on I first navigated between a few different pages. Each page load took between 5 and 15 seconds by my estimates, which is my major issue day to day. I then did a fast refresh. This was nice and quick, better than usual. To get things ready for submitting this test I upgraded to webpack 5 from 4, which might explain the improvement. Thanks again!

... I am having trouble sending you the files on twitter as they are an unsupported file type, including zip. Is there another way I can share the trace with you as I don't want to post my config files here.

[timneutkens]

@AdamDiament I've replied to your message on twitter and also updated the instructions here. https://gist.github.com is how you can share individual files 👍

[ValentinH]

I had a similar issue and it was also due to @material-ui/icons and loading all of them. If you import them directly from your application code, the tree-shaking works properly. However, in our case, we were importing them from an internal package installed in the node_modules of our application which resulted in all icons being imported (only in the server bundle).

We solved it by following this documentation and adding a babel plugin to automatically rename our named imports to default imports when building and publishing our internal package.

[AdamDiament]

@AdamDiament I've replied to your message on twitter and also updated the instructions here. https://gist.github.com is how you can share individual files 👍

Thanks Tim sent you a gist on twitter

[balazsorban44]

@stevensturkop checking in, what's the status here? Were you able to reach out to Tim? Is this issue resolved?

[Sameedahmad]

I am having this issue to be honest it sucks to wait for so long on development.

[timneutkens]

@Sameedahmad see https://github.com/vercel/next.js/issues/29559#issuecomment-938431883. Please provide a trace file.

[Sameedahmad]

@timneutkens Here are the files, https://gist.github.com/Sameedahmad/68271c9bf81883e4014f3ac3cbc32648 Let me know if you would need anything else from my side.

[timneutkens]

nextjs-wordpress-starter\functions\wordpress\blocks\displayBlock.js is taking a long time to compile.

It consists of:

• nextjs-wordpress-starter\components\blocks\Gutenberg\BlockGravityForm\index.js
• nextjs-wordpress-starter\components\atoms\Code\Code.js
• react-syntax-highlighter

Seems that react-synax-highlighter seems to be compiling support for every language in existence for your case.

Besides that it seems that reading files from the filesystem is slow in your case 🤔 Given that you have no customization in babelrc you can remove the babelrc to leverage the new compiler: https://nextjs.org/docs/advanced-features/compiler

[dottodot]

@timneutkens I've sent you my trace on twitter

[mustafa-hanif]

@timneutkens I've added my trace here. https://gist.github.com/mustafa-hanif/8deae35eb4113684c3ace37fbc610568

[timneutkens]

@mustafa-hanif I had a look at your trace and it's the same issue as the others, you're importing all of the material-ui icons library (5.5K of them) in one module. That module seems to be packages/hotels/src/ui/src/Dialog/DialogWithHeading.js.

└─ module /Users/mustafa.hanif/elaa-web/packages/hotels/src/ui/src/Dialog/DialogWithHeading.js 1s (total 281 ms, self 21 ms) [next-swc-loader 19 ms] └─ module @mui/icons-material (@mui/icons-material/esm/index.js + 9899) 260 ms (self 🔥327s) [read-resource 🔥325s]

[jason-yolabs]

~~Is there a public tool available to parse the .next/trace file? Would love to dive deeper without bugging Tim :)~~

Edit: https://github.com/vercel/next.js/blob/canary/scripts/trace-to-tree.mjs is the tool (and it's awesome!)

[Carduelis]

I've experienced the issue with @mui/material and my own npm package @gorazdo/tomui The problem comes from a lot of modules to process (11k) like in this article Unfortunately, this solution doesn't work. The problem is the regex (which is a completely mysterious thing for a non-rust developer. I've even tried Rust Regex online tool ), with no luck.

My Solution (3-5 times faster)

decreased 3-5 times dev build/rebuild/fast-refresh time decreased amount of modules by 10 times from 11k to 1k modules.

I've managed it using the following config:

    modularizeImports: {
      '@gorazdo/tomui': {
        transform: '@gorazdo/tomui/{{member}}',
      },
      '@mui/material': {
        transform: '@mui/material/{{member}}',
      },
      '@mui/icons-material/?(((\\w*)?/?)*)': {
        transform: '@mui/icons-material/{{ matches.[1] }}/{{member}}',
      },
    },

But I had to update imports manually for useTheme, ThemeProvider, styled, lighten and darken in my case

from:

import { Box, lighten, Typography, styled } from '@mui/material'
// for lighten and styled the regex didn't work neither the simple {{member}} way. so it is manually changed

to

import { Typography, Box } from '@mui/material'
import { styled, lighten } from '@mui/material/styles' 

To fix my library I had to use default export for each file which is going to be modularized.

Questions

1. How to deal with this regex to make it work with @mui/material ?
2. How to deal with modularization of files that have only named exports?
3. Can we automate modularization somehow?