svgo
svgo copied to clipboard
svg
FATAL ERROR out of memory after using 2.8GB RAM (convertPathData, defaultoption)
Prozessing the olderst version of File:Germany (+districts +municipalities) location map.svg (~73MB) fails afer needing more than 2,8GB RAM (I have 16GB RAM total, and during processing always more than 8GB free)
$ svgo -i "20180507223736!Germany_(+districts_+municipalities)_location_map_2013.svg" -o output.svg
<--- Last few GCs --->
[14628:000001C5D93EBC40] 1345238 ms: Mark-sweep 1384.2 (1457.6) -> 1384.1 (1458.1) MB, 3403.4 / 0.0 ms allocation failure GC in old space requested
[14628:000001C5D93EBC40] 1347898 ms: Mark-sweep 1384.1 (1458.1) -> 1384.1 (1427.1) MB, 2659.4 / 0.0 ms last resort GC in old space requested
[14628:000001C5D93EBC40] 1350475 ms: Mark-sweep 1384.1 (1427.1) -> 1384.1 (1427.1) MB, 2576.5 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 00000248B9D257C1 <JSObject>
2: split(this=000000CF890516F9 <Very long string[12093]>)
3: fn [C:\Users\jkalliau\AppData\Roaming\npm\node_modules\svgo\plugins\convertPathData.js:~56] [pc=000001C934FF3D35](this=00000175BEC37051 <Object map = 000000B31E6A17E1>,item=0000027C01EF5009 <JSObject>,params=00000175BEC37E89 <Object map = 000000B31E6BA439>)
4: arguments adaptor frame: 3->2
5: /* anonymous */(aka /* an...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node_module_register
2: v8::internal::FatalProcessOutOfMemory
3: v8::internal::FatalProcessOutOfMemory
4: v8::internal::Factory::NewRawTwoByteString
5: v8::internal::Smi::SmiPrint
6: v8::internal::StackGuard::HandleInterrupts
7: v8::internal::RegExpImpl::Exec
8: v8::internal::RegExpImpl::Exec
9: v8::internal::interpreter::BytecodeArrayRandomIterator::UpdateOffsetFromIndex
10: 000001C934E843C1
You can try replacing svgo with NODE_OPTIONS=--max_old_space_size=8192 node /usr/local/bin/svgo. I had to process a 120MB file which took 5GB of RAM.
@hansottowirtz Thanks for letting me know.
But sadly i am using Windows 10.
cmd.exe NODE_OPTIONS=--max_old_space_size=8192 node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg
leads to (German)
Der Befehl "NODE_OPTIONS" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
which means (English)
The command "NODE_OPTIONS" is either misspelled or could not be found
cygwin: NODE_OPTIONS=--max_old_space_size=8192 node /cygdrive/c/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg
leads to
module.js:549
throw err;
^
Error: Cannot find module 'C:\cygdrive\c\Users\jkalliau\AppData\Roaming\npm\svgo'
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Function.Module.runMain (module.js:693:10)
at startup (bootstrap_node.js:188:16)
at bootstrap_node.js:609:3
cygwin: NODE_OPTIONS=--max_old_space_size=8192 node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg
leads to
C:\Users\jkalliau\AppData\Roaming\npm\svgo:2
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
^^^^^^^
SyntaxError: missing ) after argument list
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:616:28)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Function.Module.runMain (module.js:693:10)
at startup (bootstrap_node.js:188:16)
at bootstrap_node.js:609:3
MINGW64: leads to
bash: node: Kommando nicht gefunden.
(bash: node: commando not found)
For Windows you have to set an environment variable like this:
set ENV_VAR=value
So this could become:
set NODE_OPTIONS=--max_old_space_size=8192 && node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg
@hansottowirtz Thanks, but that still didn't solve my problem
in cmd.exe: set NODE_OPTIONS=--max_old_space_size=8192 && node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg
C:\Users\jkalliau\AppData\Roaming\npm\svgo:2
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
^^^^^^^
SyntaxError: missing ) after argument list
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:616:28)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Function.Module.runMain (module.js:693:10)
at startup (bootstrap_node.js:188:16)
at bootstrap_node.js:609:3
I installed svgo from Cygwin, maybe I have to deinstall it and install it from cmd.exe
Even
@hansottowirtz Thanks, but that still didn't solve my problem
in cmd.exe:
set NODE_OPTIONS=--max_old_space_size=8192 && node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svgC:\Users\jkalliau\AppData\Roaming\npm\svgo:2 basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')") ^^^^^^^ SyntaxError: missing ) after argument list at createScript (vm.js:80:10) at Object.runInThisContext (vm.js:139:10) at Module._compile (module.js:616:28) at Object.Module._extensions..js (module.js:663:10) at Module.load (module.js:565:32) at tryModuleLoad (module.js:505:12) at Function.Module._load (module.js:497:3) at Function.Module.runMain (module.js:693:10) at startup (bootstrap_node.js:188:16) at bootstrap_node.js:609:3I installed svgo from Cygwin, maybe I have to deinstall it and install it from cmd.exe
Did you got this working ? , i have installed from node command prompt but it still does not work and throws error your mentioned above In my case i have just 30mb svgo file which i am trying to optimize though command prompt command.. and it still throws " javascript heap out of memory" exception
Did you got this working ? , i have installed from node command prompt but it still does not work and throws error your mentioned above
It still does not work on Windows, with the error-messages above
I tried on Ubuntu:
#!/bin/bash
wget https://upload.wikimedia.org/wikipedia/commons/archive/2/2e/20180507223736%21Germany_%28%2Bdistricts_%2Bmunicipalities%29_location_map_2013.svg -O input.svg
svgo -i input.svg -o output.svg
and I get:
#
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#
And the Comment from @hansottowirtz leads to
wget https://upload.wikimedia.org/wikipedia/commons/archive/2/2e/20180507223736%21Germany_%28%2Bdistricts_%2Bmunicipalities%29_location_map_2013.svg -O input.svg
NODE_OPTIONS=--max_old_space_size=8192
/usr/bin/svgo -i input.svg -o output2.svg
and results on Ubuntu in
<--- Last few GCs --->
[10799:0x2650db0] 580183 ms: Mark-sweep 1380.5 (1459.1) -> 1380.5 (1460.1) MB, 1004.1 / 0.0 ms allocation failure GC in old space requested
[10799:0x2650db0] 581383 ms: Mark-sweep 1380.5 (1460.1) -> 1380.5 (1429.1) MB, 1199.7 / 0.0 ms last resort GC in old space requested
[10799:0x2650db0] 582150 ms: Mark-sweep 1380.5 (1429.1) -> 1380.5 (1429.1) MB, 766.9 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2aa043257c1 <JSObject>
1: /* anonymous */(aka /* anonymous */) [/usr/lib/node_modules/svgo/plugins/convertPathData.js:~652] [pc=0x2dd33817bb47](this=0x1881e95822d1 <undefined>,item=0x249aafffbc71 <Object map = 0x2925d02feb11>,index=54)
2: arguments adaptor frame: 3->2
3: filter(this=0x3f02cb2de551 <JSArray[146]>)
4: fn [/usr/lib/node_modules/svgo/plugins/convertPathData.js:~59] [pc=0x2dd33815a1f8](this=...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x11e7fec [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [node]
6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
7: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
8: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x2dd337f842fd
Aborted (core dumped)
@JoKalliauer
I believe you need to set the environment variable for NODE_OPTIONS.
On Windows via:
set NODE_OPTIONS=--max_old_space_size=8192
On Linux via:
export NODE_OPTIONS=--max_old_space_size=8192
Alternatively, you can try svgcleaner. I found this reduces size more and is extremely fast, as it is written in Rust.
I managed to make this compress on Windows with Node.js 14.15.5 and
C:\Users\xmr\Desktop>set NODE_OPTIONS=--max_old_space_size=8192 && svgo --pretty 1.svg -o 2.svg
1.svg:
Done in 172078 ms!
74491.22 KiB - 49.3% = 37762.557 KiB
That being said, maybe there are some improvements that can be made to reduce memory allocations?
@TrySound might be worth documenting the workaround for all OS'es in README for such special cases. The above will work on Windows.
I'm pretty sure we can improve the situation a bit later, but I'm not sure how much.
BTW @TrySound 2.1.0 vs 2.2.2 has a huge speed penalty, at least on my Windows VM:
C:\Users\xmr\Desktop>npm i svgo@~2.1 -g
C:\Users\xmr\AppData\Roaming\npm\svgo -> C:\Users\xmr\AppData\Roaming\npm\node_modules\svgo\bin\svgo
+ [email protected]
added 1 package from 1 contributor and updated 1 package in 1.848s
C:\Users\xmr\Desktop>set NODE_OPTIONS=--max_old_space_size=8192 && svgo 1.svg -o 2.svg
1.svg:
Done in 164252 ms!
74491.22 KiB - 49.3% = 37749.765 KiB
C:\Users\xmr\Desktop>npm i svgo@latest -g
C:\Users\xmr\AppData\Roaming\npm\svgo -> C:\Users\xmr\AppData\Roaming\npm\node_modules\svgo\bin\svgo
+ [email protected]
removed 1 package and updated 1 package in 1.521s
C:\Users\xmr\Desktop>set NODE_OPTIONS=--max_old_space_size=8192 && svgo 1.svg -o 2.svg
1.svg:
Done in 355186 ms!
74491.22 KiB - 49.3% = 37737.378 KiB
We really need a benchmark script. BTW the speed issue is noticeable with smaller files too, but it's not so huge.
https://github.com/svg/svgo/compare/v2.1.0...v2.2.2?w=1
Now, I'm not sure which commit is to blame yet. I could try to pinpoint it in the next days unless someone else beats me to it.
I suggest that you finish with the prettier stuff so that it doesn't pollute the git history anymore; it should have been made in one commit. And also add npm scripts to check and format.
I'll close this issue as it's quite open-ended, and we're actively trying to improve the performance of SVGO as we go. I haven't personally been able to reproduce it under the same conditions, but appreciate that given a large enough SVG or low enough available memory this will still occur. My hope is that the situation here will improve over the next few releases.
Here are some stats showing how SVGO is performing with the main branch and other recent releases:
| Version | Time | Reduction | RAM Usage |
|---|---|---|---|
main (8d6385bd9ab49d1d300a10268930238baa5eb269) |
46585 ms | 49.5% | 2.827~ GB |
| v3.2.0 | 50651 ms | 49.4% | 2.856~ GB |
| v3.1.0 | 74173 ms | 49.4% | 3.166~ GB |
| v3.0.5 | 72965 ms | 49.4% | 3.225~ GB |
| v3.0.4 | 75558 ms | 49.4% | 3.187~ GB |
Overall, time has significant improved, memory usage has marginally improved, and compression has negligibly improved.
The rationale for the time and memory improvements are mostly for https://github.com/svg/svgo/pull/1918, and once merged will be used as the benchmark to ensure we understand the performance implications of new contributions.
Meanwhile, we may explore solutions like chunking large SVG files or clearing caches when we're using too much memory, etc. That would undermine speed ofc, but may improve stability if users are having OOM issues.
Rather than take action, since it's unclear when we've done enough, we'll just ensure all future work keeps memory/performance in mind, and that we profile SVGO and upstream dependencies to see where we can improve.
I retried it on another computer, and I can't reproduce the bug either. I consider it as fixed or not reproducable report. Thanks for your efford.
Edit: Others can still reproduce the bug.
I can certainly reproduce the issue:
C:\Users\xmr\Desktop>curl https://upload.wikimedia.org/wikipedia/commons/archive/2/2e/20180507223736%21Germany_%28%2Bdistricts_%2Bmunicipalities%29_location_map_2013.svg>input.svg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 72.7M 0 72.7M 0 0 5669k 0 --:--:-- 0:00:13 --:--:-- 5544k
C:\Users\xmr\Desktop>npm i -g svgo
added 19 packages in 1s
9 packages are looking for funding
run `npm fund` for details
C:\Users\xmr\Desktop>svgo -i input.svg -o output.svg
<--- Last few GCs --->
[1448:00000169B97559B0] 8658 ms: Scavenge (reduce) 2047.2 (2081.9) -> 2047.2 (2082.7) MB, 4.01 / 0.00 ms (average mu = 0.169, current mu = 0.109) allocation failure;
[1448:00000169B97559B0] 8964 ms: Mark-Compact (reduce) 2048.1 (2082.7) -> 2048.1 (2083.7) MB, 230.03 / 0.00 ms (+ 143.9 ms in 29 steps since start of marking, biggest step 5.4 ms, walltime since start of marking 389 ms) (average mu = 0.221, current m
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 00007FF7025B71AB node::SetCppgcReference+16075
2: 00007FF70252DCC6 v8::base::CPU::num_virtual_address_bits+79190
3: 00007FF70252FED5 v8::base::CPU::num_virtual_address_bits+87909
4: 00007FF702F9F061 v8::Isolate::ReportExternalAllocationLimitReached+65
5: 00007FF702F887F8 v8::Function::Experimental_IsNopFunction+1336
6: 00007FF702DEA120 v8::Platform::SystemClockTimeMillis+659328
7: 00007FF702DE71A8 v8::Platform::SystemClockTimeMillis+647176
8: 00007FF702DFC4BA v8::Platform::SystemClockTimeMillis+733978
9: 00007FF702DFCD37 v8::Platform::SystemClockTimeMillis+736151
10: 00007FF702E0B62F v8::Platform::SystemClockTimeMillis+795791
11: 00007FF702ACBBE5 v8::CodeEvent::GetFunctionName+116773
12: 00007FF6A301AAFA
C:\Users\xmr\Desktop>node -v
v20.11.1
If you increase the old space size to 4000, it works fine.
➜ dir NODE_OPTIONS=--max_old_space_size=8192 pnpm exec svgo -i input.svg -o output.svg
input.svg:
Done in 67604 ms!
74491.22 KiB - 49.4% = 37659.443 KiB
➜ dir NODE_OPTIONS=--max_old_space_size=4000 pnpm exec svgo -i input.svg -o output.svg
input.svg:
Done in 106948 ms!
74491.22 KiB - 49.4% = 37659.443 KiB
I'm not sure if it's worth the time to try to decrease the amount of RAM used on giant files.
Yeah, I'm aware I just to make it clear that the issue does exist and it's not solved.
I can certainly reproduce the issue:
Thanks for sharing, I figured on some environments it'd still be possible.
I'm not sure if it's worth the time to try to decrease the amount of RAM used on giant files.
In my opinion, it is worth the time. It's not top priority, but making SVGO work with large files and be better equip to operate on smaller environments like phones or CI is favorable.
For example, I wouldn't want OhMySVG crashing on my phone (PinePhone/Linux) because SVGO is using too much memory. If we can find better ways to store and process data that reduces our memory footprint, it's just better for everyone.
Allocating memory takes time, which we're trying to get down, and reserving resources that other processes or the system memory cache could be using instead isn't considerate of our users or the other software installed on their machine.
TL;DR: Large files not being our primary use case, or the assumptions that clients have the resources to spare, isn't a good reason not to try to use resources more efficiently.
Edit: Sorry I'm less active right not btw, I'm sick. 😷