heap out of memory in a monorepo with --deep

[simllll]

• [x] I have searched for similar issues
• [x] I am using the latest version of npm-check-updates
• [x] I am using node >= 12

---

Steps to Reproduce

I have a monorepo wiht > 100 packages in it, if I run the following command: npx ncu -x marked,csv-stringify,@hokify/node-ts-cache*,dateformat,node-fetch,p-queue,vue-i18n,autoprefixer,ts-loader,sass-loader,postcss,ejs,webpack,@elastic/elasticsearch -u --deep --timeout 86400000 -t minor it crashes after some time. I increased the memory already up to 16gigabyte (with max old space paramter), it crashes later, but it still crashes.. therefore I assume that must be some kind of a bug?

Current Behavior

[======--------------] 11/36 30%
<--- Last few GCs --->

[38817:0x4d89b60]   850393 ms: Mark-sweep 7980.7 (8137.8) -> 7960.6 (8133.1) MB, 1877.8 / 0.0 ms  (average mu = 0.142, current mu = 0.033) allocation failure scavenge might not succeed
[38817:0x4d89b60]   852279 ms: Mark-sweep 7981.0 (8142.8) -> 7979.7 (8152.5) MB, 1812.2 / 0.0 ms  (average mu = 0.094, current mu = 0.039) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node]
 3: 0xcf77be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcf7b37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xeaf3d5  [node]
 6: 0xebf09d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xec1d9e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xe83012 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
 9: 0xe7d92c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
10: 0xe7da05 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
11: 0x10e0dc3 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::NewInternal<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType) [node]
12: 0x10e10b3 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::AllocationType) [node]
13: 0x10e1116 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [node]
14: 0x10e177a v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [node]
15: 0x10aea29 v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>) [node]
16: 0x10cddeb v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [node]
17: 0x1084089 v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::JSObject::AccessorInfoHandling) [node]
18: 0xf9f290 v8::internal::JsonParser<unsigned char>::BuildJsonObject(v8::internal::JsonParser<unsigned char>::JsonContinuation const&, v8::base::SmallVector<v8::internal::JsonProperty, 16ul> const&, v8::internal::Handle<v8::internal::Map>) [node]
19: 0xfa04f5 v8::internal::JsonParser<unsigned char>::ParseJsonValue() [node]
20: 0xfa124f v8::internal::JsonParser<unsigned char>::ParseJson() [node]
21: 0xd79808 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
22: 0x15f0b79  [node]
Aborted (core dumped)
simon@simon-ThinkPad-X1-Carbon-Gen-9:~/Dev/hokify/hokify-server/packages$ 
simon@simon-ThinkPad-X1-Carbon-Gen-9:~/Dev/hokify/hokify-server/packages$ 
simon@simon-ThinkPad-X1-Carbon-Gen-9:~/Dev/hokify/hokify-server/packages$ 
simon@simon-ThinkPad-X1-Car

Expected Behavior

no crashes ;-)

If it helps, I can upload the core dump somewhere?

[raineorshine]

Yes, the --deep algorithm runs a reducer on all subpackages, and is not memory-optimized. I don't have the bandwidth to investigate at this time, but thank you for reporting. The performance could definitely be improved to handle large monorepos.

Here's the reducer that calls runLocal for each package it finds:

https://github.com/raineorshine/npm-check-updates/blob/896cc1178cd7ad1e231cb14527b615dcb94742dc/src/index.ts#L85-L115

As a workaround (not ideal, but just so you have it), you could write a shell script that calls ncu separately for each subpackage. It might be able to get away with calling ncu --deep on the subpackages that are just one level down.