RAM comsumption issue when trying to limit the usage

[curquiza]

We need to investigate the RAM consumption issue we have with the flag to limit consumption (for big datasets).

Sorry my issue is not clear, we are going to complete it as we go along.

Not needed for the first release of the new search engine.

(Also Kero sorry if you already created one 😇)

[curquiza]

Another related issue of this RAM usage: during the indexation of a huge dataset, Milli uses a lot of RAM (ex: 8G). But when the indexation is over, Milli still uses the same amount of RAM during the search (ex: still 8G). If you stop and restart Milli, the usage of RAM during the search is correct again (around 2M for example).

[curquiza]

@Kerollmops and @irevoire investigated it. It seems coming from LMDB that creates an internal linked list (certainly to optimize quick operations) but this list is not free and is responsible for this RAM consumption even after the indexation is over. @Kerollmops will open an issue in LMDB to suggest solutions to remove/reduce the impact of this linked list.

⚠️ Even if LMDB is updated, we still need to wait for the update of LMDB in mozilla/lmdb-rs.

[homersimpsons]

It seems that mozilla/lmdb-rs is getting replaced by https://github.com/mozilla/rkv

[curquiza]

RKV is a wrapper around lmdb-rs so it does not remove our issue I think.

[GregoryConrad]

This issue was fixed over in https://github.com/meilisearch/lmdb-rs/pull/1, right? Or is that a different issue?

[curquiza]

This issue is too old, and yes already fixed a long time ago! thanks @GregoryConrad