Memory leak is observed when using decache

[hjain60]

I am using decache to re-require modules in loop. The server crashes as memory leak is observed.

call the following code recursively and you will observe significant memory leak:

var file = 'some file';
scheduleTimer = setInterval(function () {
      updateFileCache();
    }, pullTime);

updateFileCache: function (file) {
    decache(file);
    var newPlugin = require(file);
    return newPlugin;
  }

[nelsonic]

@hjain60 thanks for opening this issue. What is your use case that necessitates re-requiring and decaching a module in a loop? 🤔

[hjain60]

This is for a plugin service where modules from users are hot loaded. If changes are made to module, the modules are re-loaded again.

User can make any number of changes and each change triggers the re-requiring of module and its children module.

[runi95]

Had the same issue when using decache to hot load custom configuration in a dev environment.

[nelsonic]

Sadly, this is a Node.js issue not a decache one. The decache module is simply deleting the cached (imported/required) modules.

https://github.com/dwyl/decache/blob/501de25900f0a4c68aa8c388bc655f2d05e7ee26/decache.js#L35

If Node.js/V8 is not doing garbage collection on the deleted data, there's not much we can do about it.

🤷‍♂

Obviously if someone wants to use the --expose-gc and force Node.js/V8 to do garbage collection on the require cache, we would love to get a Pull Request adding that functionality to decache.

The Pull Request just needs a couple of Unit Tests, Docs (informing people to run their node app/script with the --expose-gc flag) and an if statement to invoke the garbage collection. e.g:

if (global.gc) { global.gc(); }

If you need this now and don't have time to create a PR in decache, simply do the following:

1. run your app using the --expose-gc flag e.g: node server.js --expose-gc
2. in each place you are using decache('mymodule.js'), add the line: if (global.gc) { global.gc(); }

That will avoid memory leaks because all garbage (deleted cache) will be cleared each time.

If you want to understand this in more detail, please read: “Understanding Garbage Collection and Hunting Memory Leaks in Node.js” by @danielkhan: https://blog.codeship.com/understanding-garbage-collection-in-node-js

[runi95]

I created a test file that just decaches and then requires the same module repeatedly and then finally goes to sleep for a minute:

const decache = require("decache");

const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

const run = async () => {
    const N = 1000;
    const dir = "../examples";
    for (let i = 0; i < N; i++) {
        decache(dir);
        require(dir);
    }

    console.log("Going to sleep...");
    await sleep(60000);
};

run();

It starts off using 2.1 MB RAM, peaks at 4.4 MB and then settles at 4.3 MB before terminating.

Replacing decache with clear-module makes this test program peak at 2.2 MB and terminate at 2.2 MB. I ran it with node --inspect src and monitored it using the built in inspector tool in Chrome.

My Node version is v13.7.0 My decache version is 4.5.1 My clear-module version is 4.1.0

[nelsonic]

@runi95 thanks for sharing. 👍 TODO: read the source of clear-module and compare to decache.js 🔍

[MCArth]

Given this still hasn't been fixed, it should be mentioned in the README to prevent people from running into it over and over again

[nelsonic]

@MCArth thank you for your comment. 👍 If you want to PR the fix please go for it. 🙏 Someone of your skill level can probably figure this out before breakfast. 🧑‍💻 I've granted you write access to the repo: https://github.com/dwyl/decache/invitations

This is a very niche edge-case. decache is a useful devDependency but we've never seen a need for it beyond testing. How many people are de-caching a node_module and re-importing it in a loop or setInterval in practice? 🤷‍♂️

Not saying the issue isn't valid there is definitely a "leak" albeit a tiny one unless someone is importing a mega library. I wish I had time to fix it ... (PRs very much welcome! 🙏 ) If it were something that people were likely to see in production, it would totally make sense to have an <h1> warning in the README.md.