Memory leak w/ cache introduced in 2.2.1?

[leiffoged]

First of all, thanks for making this awesome library.

I'm running into an issue where memory seems to grow uncontrollably.

My test (Default options to new DB except with cache: set to true):

• Add 5,000,000 small items (e.g. 16 byte keys/values) to the database
• Clear the database (using clearAsync)
• Repeat

On versions prior to 2.2.1 (e.g. 2.2.0 and before), nodeJS memory seems to stabilize (for my application, that's around 200-300MB), even after running the above loop for a very long time.

On versions starting on or after 2.2.1, however, memory usage grows uncontrollably and within a few minutes my process is using several gigabytes of memory.

I've worked around this issue by simply disabling the cache.

Known issue?

[kriszyp]

No, this isn't a known issue, and there wasn't a lot that changed between 2.2.0 and 2.2.1 (https://github.com/DoctorEvidence/lmdb-js/compare/v2.2.0...v2.2.1), the main things were some changes to how compression is instantiated (you aren't using compression?), handling of large buffers (sounds like you are seeing this with small values), and cleanup when closing.

In that timeframe there was an update to the weak-lru-cache package that is used, but that was to reduce retained objects, and shouldn't cause increased memory usage.

Do you ever experience out of memory crashes? Or does it eventually stabilize at several gigabytes? V8 certainly is pretty non-deterministic about how much memory it allocates and how long it goes until it GCs.

Here is the code I tried to attempt to reproduce this:

      while(true) {
        for (let i = 0; i < 5000000; i++) {
          db.put(i, i);
        }
        await db.committed;
        await db.clearAsync();
      }

But the memory usage is basically the same across v2.2.0, v2.2.1 and latest, shifting back and forth between about 1.5GB and 3GB.

Do you have any code snippets to reproduce this?

[kriszyp]

While I haven't found any regressions, I did actually add some optimizations for clearing memory better for large number of writes that occur in a single event turn/txn (as in my example code). In my example code this significantly reduced memory usage during heavy writes. This is published in v2.3.0-beta3.