JSCPD 3.3.19 throws the exception: JSCPD ERROR OUTPUT: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[ghost]

Describe the bug JSCPD 3.3.19 throws the below exception when scanning larger archives. JSCPD ERROR OUTPUT: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

To Reproduce Steps to reproduce the behavior:

1. Command used: jscpd <PATH_TO_SCAN> --reporters html,json --min-tokens 50 --format csharp,typescript,javascript,java,cpp,c --ignore /*.min.js,/*.map,/node_modules/,/Properties/,/bin/,/obj/ --output <Path_To_Report> -z 1000mb -x 1000000

Output:

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x033b035808d1 <JSObject> 0: builtin exit frame: concat(this=0x03f7eeff0fa9 <JSArray[244]>,0x03f7eeff0fc9 <JSArray[1]>,0x03f7eeff0fa9 <JSArray[244]>)

1: /* anonymous */(aka /* anonymous */) [000003F7EEFED261] [C:\Users\<AppData\Roaming\npm\node_modules\jscpd\node_modules\@jscpd\tokenizer\dist\tokenize.js:108] [bytecode=00000033C63AD991 offset=28](this=0x01e47f9c04b1 <undefined>,0x03f7eefe61d9 <Token map...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 00007FF682555F0F napi_wrap+114095 2: 00007FF682500B96 v8::base::CPU::has_sse+66998 3: 00007FF682501996 v8::base::CPU::has_sse+70582 4: 00007FF682D16E9E v8::Isolate::ReportExternalAllocationLimitReached+94 5: 00007FF682CFEF71 v8::SharedArrayBuffer::Externalize+833 6: 00007FF682BCB1DC v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436 7: 00007FF682BD6410 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312 8: 00007FF682BD2F34 v8::internal::Heap::PageFlagsAreConsistent+3204 9: 00007FF682BC8733 v8::internal::Heap::CollectGarbage+1283 10: 00007FF682BC6DA4 v8::internal::Heap::AddRetainedMap+2452 11: 00007FF682BE822B v8::internal::Factory::NewFixedArrayWithFiller+107 12: 00007FF682BE9D42 v8::internal::Factory::NewJSArrayStorage+210 13: 00007FF682BE970A v8::internal::Factory::NewJSArray+154 14: 00007FF682AB3E85 v8::internal::FeedbackNexus::ic_state+58149 15: 00007FF682CC6AAE v8::internal::Builtins::builtin_handle+306942 16: 00007FF682CC4712 v8::internal::Builtins::builtin_handle+297826 17: 00007FF682CC434E v8::internal::Builtins::builtin_handle+296862 18: 00007FF68316B59D v8::internal::SetupIsolateDelegate::SetupHeap+546893 19: 00007FF6830F044C v8::internal::SetupIsolateDelegate::SetupHeap+42748 20: 00007FF68317D68C v8::internal::SetupIsolateDelegate::SetupHeap+620860 21: 00007FF6830F044C v8::internal::SetupIsolateDelegate::SetupHeap+42748 22: 00007FF6830F044C v8::internal::SetupIsolateDelegate::SetupHeap+42748 23: 00007FF6830F044C v8::internal::SetupIsolateDelegate::SetupHeap+42748 24: 00007FF6830F044C v8::internal::SetupIsolateDelegate::SetupHeap+42748 25: 00007FF68311DFB5 v8::internal::SetupIsolateDelegate::SetupHeap+229989 26: 0000014E15B43DC3

Desktop (please complete the following information):

• OS: Windows 10 Enterprise
• NodeJS Version v12.18.4
• jscpd version 3.3.19

Additional context I have so far tried the following:

1. Tweak the excludes as necessary based on the point of occurrence of the exception. This may work sometime – but, is challenging with PR’s because you can have files added/modified across the whole archive. As a result ,it is not very predictable at the moment.

2. Setting the space size: node --max-old-space-size=8192 node_modules\jscpd\bin\jscpd

3. Removing the .jscpd directory before execution and a force clean of the npm cache: npm cache clean –force

4. Increasing the –z –x values, it works sometimes – not predictable.

5. Removing –z –x options, it works sometimes – not predicatble.

None of the above have worked consistently.

[kucherenko]

Looks like you have a really huge code base, try to use leveldb store.

$ npm install @jscpd/leveldb-store

$ jscpd --store leveldb /pathtocode

[ghost]

Hi, When I use the jscpd --store leveldb, I get the following exception only for cpp - the other formats work fine.

Detection time:: 714.933ms Error: EBUSY: resource busy or locked, unlink '.jscpd\cpp\000003.log' at Object.unlinkSync (fs.js:1136:3) at rimrafSync (C:\Users<USER>\node_modules\rimraf\rimraf.js:303:17) at C:\Users<USER>\node_modules\rimraf\rimraf.js:336:39 at Array.forEach () at rmkidsSync (C:\Users<USER>\node_modules\rimraf\rimraf.js:336:26) at rmdirSync (C:\Users<USER>\node_modules\rimraf\rimraf.js:329:7) at rimrafSync (C:\Users<USER>\node_modules\rimraf\rimraf.js:301:9) at C:\Users<USER>\node_modules\rimraf\rimraf.js:336:39 at Array.forEach () at rmkidsSync (C:\Users<USER>\node_modules\rimraf\rimraf.js:336:26) { errno: -4082, syscall: 'unlink', code: 'EBUSY', path: '.jscpd\cpp\000003.log'

[kucherenko]

is the error at the end of report?

[ghost]

Yes - that is correct.