App takes long to end after work is done

[jonim8or]

I created this custom script based on the samples:

const {detectClones} = require("jscpd");
const {MemoryStore} = require("@jscpd/core");

const BASE_APPS = [
//several source folders here
];
const DEFAULT_OPTIONS = {
    path: BASE_APPS,
    silent: true,
    minLines: 15,
    minTokens: 60,
    output: './master-report-js',
    reporters: ['json'],
    format: 'javascript,css,less,markup'
};

(async () => {
    const store = new MemoryStore();
    await detectClones(DEFAULT_OPTIONS, store);
})();

For me it takes about 6 seconds to output:

JSON report saved to master-report-js\0\jscpd-report.json Detection time:: 5916.966ms

After that it takes about 20 seconds more... and it is unclear to me what it is doing. It seems to happen after the await returns, because adding console.logs in a .then() clause also makes them also print out directly after the detection time, but then it takes 20 seconds more.

My quick and dirty solution is to add this to the detectClones... but it feels really horrible

await detectClones(DEFAULT_OPTIONS, store).then(()=>process.exit(0));

Could it be that the detectClones creates lots of events / async calls that need to be cleaned up?

The following turned out to be caused by a mistake I made in the file paths with the above program,so please ignore the remark below. ~~Running the same command using the jscpd main app takes significantly longer:~~

npx jscpd --output ./master-report-double/0 --format javascript,css,less,markup --min-tokens 60 --min-lines 15 --ignore *.stories.js --silent --reporters json [...lots of source dirs]
JSON report saved to master-report-double\0\jscpd-report.json
Duplications detection: Found 86 exact clones with 3507(4.59%) duplicated lines in 596 (4 formats) files.
Detection time:: 261734.781ms

[kucherenko]

Hm... looks interesting, will investigate

[jonim8or]

Small update. The extremely large detection time in the last command was due to a typo in one of the file paths... so I will cross out the last part. But the remark in the rest of my message remains: I don't understand why after the end of my "program" it takes so long for the application to return. After a jscpd test that has a detection time of 271500 ms (4.5 min), it takes an additional 2 minutes until it is completed A run with the jscpd command line takes about the same time (287780ms) but does not have the additional 2 minutes

[jonim8or]

I created a simple way to reproduce it. It checks out moment.js and tries to find duplicates there. jscpd-411.zip

const {detectClones} = require("jscpd");

(async () => {
    console.time('Total time');
    await detectClones({path:['moment/src'], silent: true, output:'jscpd-output'});
    console.timeEnd('Total time');
	//This is where the ugly hack would come in 
	//process.exit(0);
})();

#!/bin/sh
if [ ! -d "/home/node/moment" ]
then 
	git clone https://github.com/moment/moment.git
fi
npm install
echo "Starting execution of main script"
START_TIME=$(date +%s)
node main.js
END_TIME=$(date +%s)
echo "Execution took $(($END_TIME - $START_TIME)) seconds"

It gives this output (when run with node 15)

Cloning into 'moment'...
remote: Enumerating objects: 28136, done.
remote: Total 28136 (delta 0), reused 0 (delta 0), pack-reused 28136
Receiving objects: 100% (28136/28136), 19.51 MiB | 5.51 MiB/s, done.
Resolving deltas: 100% (19477/19477), done.
Checking out files: 100% (786/786), done.
npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.

added 130 packages, and audited 130 packages in 16s

10 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New patch version of npm available! 7.0.8 -> 7.0.13
npm notice Changelog: https://github.com/npm/cli/releases/tag/v7.0.13
npm notice Run npm install -g [email protected] to update!
npm notice
Starting execution of main script
Found 1302 clones.
Detection time:: 24.333s
Total time: 26.398s
Execution took 102 seconds

[kucherenko]

thank you!

[JiangWeixian]

same issue

[fairytalexw]

same issue, how to fix~

[sonacy]

any updates?

[sonacy]

when you changed memory store to sth like leveldb-store, the problem will be fixed. the key problem is about using promise in recursive maybe causing memory leak. I tried several ways to fix that.

1. you can just changed the file rabin-karp.ts, changes the loop function.

.finally(() => {
    if (!iteration.done) {
      if (clone) {
        clone = RabinKarp.enlargeClone(
          clone,
          iteration.value as IMapFrame,
          mapFrameInStore
        );
      }
      // loop
      setImmediate(loop);
    } else {
      resolve(clones);
    }
  });

i just change loop to setImmediate(loop), in that way, the event loop will not create a new task, but just execute the loop as a micro task immediate. but it will cause more time and memory to handle.
2. I just changed store to plain function， without using promise。

for same test code, first way causing 20s, but second causing 9s.

[kucherenko]

@sonacy thank you for your investigation, will try to fix it, or can merge you your PR