kit
kit copied to clipboard
sveltejs
Blocked migration to SvelteKit 2 - Out of memory & explosion in compilation time
Describe the bug
We have decided to try migrating our relatively large app to SvelteKit 2 (dashboard with tons of components and [internal] dependencies).
Unfortunately, we have to give up upgrading.
The amount of memory required for the build process has doubled. We now need a server with 12GB of RAM at least to compile. Worse, the compilation time has exploded now (about 30x longer on Apple M1 Pro). This is extremely expensive.
The reasons are not SvelteKit per se, we think. But rather Vite: vitejs/vite#2433
It might be that Vite 5 has more memory leaks.
For now, we are continuing with SvelteKit 1 (and its Vite 4 dep), which has been great to us!
Reproduction
We cannot share our code base. It happens for large and complex apps.
Logs
[12305:0x138008000] 8389 ms: Mark-Compact 8138.7 (8239.1) -> 8138.5 (8239.1) MB, 79.67 / 0.00 ms (average mu = 0.511, current mu = 0.018) allocation failure; scavenge might not succeed
[12305:0x138008000] 8467 ms: Mark-Compact 8140.8 (8241.2) -> 8140.6 (8241.2) MB, 77.38 / 0.00 ms (average mu = 0.330, current mu = 0.011) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
System Info
System:
OS: macOS 14.3
CPU: (10) arm64 Apple M1 Pro
Memory: 7.89 GB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.11.0 - /opt/homebrew/bin/node
npm: 10.2.4 - /opt/homebrew/bin/npm
Browsers:
Chrome: 121.0.6167.139
Safari: 17.3
npmPackages:
@sveltejs/adapter-cloudflare: ^4.1.0 => 4.1.0
@sveltejs/kit: ^2.5.0 => 2.5.0
@sveltejs/vite-plugin-svelte: ^3.0.2 => 3.0.2
svelte: ^4.2.10 => 4.2.10
vite: ^5.0.12 => 5.0.12
Severity
blocking an upgrade
Additional Information
No response
Without a reproduction there isn't much we can do to investigate. And yes, it's very likely a Vite issue since we haven't changed things with regards to the build output. You could try to bisect it a bit by checking if earlier versions of SvelteKit 2 (2.0, 2.1 etc) make a difference, which may hint at a problem with SvelteKit after all, and you could also try earlier versions of Vite 5.
Vite has had a memory leak issue for years, and I'm afraid there's still no official fix. You can investigate if a certain dependency is making the memory leak worse and perhaps switch that dependency.
The best you can do for now is increase the maximum memory in node:
NODE_OPTIONS=--max-old-space-size=4096
or inside .npmrc add a line with max-old-space-size=4096
https://github.com/vitejs/vite/issues/2433#issuecomment-1455026901 https://support.circleci.com/hc/en-us/articles/360009208393-How-Can-I-Increase-the-Max-Memory-for-Node
Another fix is to disable sourcemaps which should greatly reduce the memory usage. https://vitejs.dev/config/build-options#build-sourcemap
there are more advaned configs for sourcemaps such as https://github.com/vitejs/vite/issues/2433#issuecomment-1482851550
Sry for the late reply.
@dummdidumm We tried a dissect on different versions of Vite 5 and Sveltekit 2, unfortunately it doesn't seem to be specific to a minor or patch version. We migrated another smaller project without impossible issues though, although memory use is indeed up.
@hichemfantar as I said, we can get it compiled by increasing Node's allocated memory (to 12GB). Disabling sourcemaps didn't change anything (the OOM might actually happen before the sourcemaps are generated).
We're considering options, and will update here.
@arbfay disabling the source maps should lower the memory usage and build time. try comparing the memory consumption and build times with source maps on and off.
We tried a dissect on different versions of Vite 5 and Sveltekit 2, unfortunately it doesn't seem to be specific to a minor or patch version.
What do you mean by that? That all versions of SvelteKit 1 + Vite 4 work but none of the Sveltekit 2 + Vite 5 combinations work? i.e. it's either SvelteKit or (more likely) Vite having a problem?
@dummdidumm exactly what I mean.
Interestingly, I've spent the last few days migrating the app to Astro 4.4 (just front-end, in dev mode), and I've just hit exactly the same issues at build time.
Astro 4 also uses Vite 5.
Pretty sure at this point that the issue is Vite 5 not being scalable at all...
Sadly I'm forced to use the JS hell-system 😭
cc @bluwy, seems like there's some regression in Vite 5 around memory consumption when building. Sadly there's no reproducible here, but maybe from looking at what was inside the Vite 4->5 jump you have a guess?
@arbfay there were some Vite 5 beta versions. Did you try to bisect those?
@hichemfantar Vite/Rollup needing large amounts of memory to build is not a memory leak. It requires keeping all modules and its representations in-memory, and that can require a large amount of memory to process, but it's not leaking anything.
Of course, there could be a leak somewhere else but it's hard to find out. @arbfay perhaps you can help measure it by running:
-
node --inspect-brk ./node_modules/vite/bin/vite.js(your build script but with node) - Open Chrome and go to
chrome://inspect - Select the inspected process (you should see your command in no1 here)
- A window will open, and you can use the "Performance" tab and the "Memory" option to record the build process.
Recording perhaps the first 1min might be enough, otherwise the generate snapshot will be very large and will take time to generate. And you can share that with us (via email maybe or a link to keep it private)
Other than that, I've not heard about Vite 5 being slower than usual. The biggest change is that Rollup moves its parser to Rust, so instead it should be slightly more efficient.
Also, while not memory-related, https://vitejs.dev/guide/performance.html may also help with debugging things.
We're experiencing similar with a larger app - https://github.com/huntabyte/shadcn-svelte.
The last build on Vite 4/SK v1:
Current builds on SK v2:
Fortunately, ours still builds, but it takes 8+ minutes vs the 2-3 we used to see. We're going to dig in to try to find out what is causing this and report back any findings.
- Note, builds are quick locally (Mac Studio), but they seem to be putting a bit of stress on the GitHub Runners/lower-end machines.
With more work, I was able to make it compile with Astro instead of SvelteKit by changing some configs (excluding node_modules scans explicitly was key to resolve the OOM).
So maybe it's not Vite 5 actually, but vite-plugin-svelte instead?
Will try to tweak configs of vite-plugin-svelte when I have more time.
cc @dominikg - anything related to building/memory consumption that was happening in the major release of v-p-s?
For https://github.com/huntabyte/shadcn-svelte, the PR that upgraded the project to SK 2 + Vite 5 also included the addition of @sveltejs/enhanced-img. Removing the plugin resulted in the build times returning back to normal (and is seemingly faster than SK 1 + Vite 4).
I'm unsure of what normal build times are expected when using this plugin, however I don't think we're experiencing the problems presented in this particular issue. Sorry for the confusion!
yes, enhanced-img uses sharp to do image transforms during build. Increased build times and memory consumption are expected in that case.
I did not observe a significant increase in memory consumption or build times in a larger project (>1000 modules > 150 first party .svelte files) going from sveltekit1 to sveltekit2, now with sveltekit 2.5 and vite 5.1 memory consumption is actually a bit lower than the last sveltekit1 build which has significantly less components. (edit: peak memory use was ~1.2GB)
We didn't have any memory related changes in v-p-s 3 either. As mentioned above this really needs more info and/or a reproduction.
Had the same issue. Can confirm that removing @sveltejs/enhanced-img resolved the issue for me
Can folks here who had issues try upgrading to @sveltejs/enhanced-img version 0.2.0? It introduces caching, which should cut build times in half on the first run and make it take no additional time on subsequent runs
Can folks here who had issues try upgrading to
@sveltejs/enhanced-imgversion 0.2.0? It introduces caching, which should cut build times in half on the first run and make it take no additional time on subsequent runs
Had the issue with version 0.2.0. I can only remember that I had heap allocation errors. I initially fixed it by increasing the heap limit. Then I removed the @sveltejs/enhanced-img without setting the heap limit and it worked
@vr7bd can you provide a repo that reproduces the issue? Unfortunately I can't. But I'll see if I can try to make a repo to replicate it. But if you need any specific info, you can let me know
We have a quite large SvelteKit app (1558 Typescript files and 757 Svelte files) and recently started getting OOM on all our builds. We upgraded to SvelteKit 2 a while ago with no problems during build, and we've also been using @sveltejs/enhanced-img for quite a while. I tried disabling @sveltejs/enhanced-img, but the OOM still happens when building.
Our current solution has been to change vite build to NODE_OPTIONS=--max-old-space-size=6144 vite build.
When you say "recently", was it a version bump that caused this, or a change in code?