Memory allocation while indexing data

[amirEBD]

Hi @valeriansaliou , I had a great experience while using sonic, but the point is while indexing lots of data into sonic, the RAM usage will grow which is so ratinale. After finishing the indexing the RAM usage won't decrease as it is like sonic holds the memory. Would you have any suggestion for that?

PS: As I delete the container and recreate it the memory usage will be about 5 MBs

[amirEBD]

Also I had an issue getting the amount of data inside the sonic. How can I get that? Cause sonic has no API for getting the volume size. Is it just using commands like sudo du data/?

[valeriansaliou]

The allocator frees its memory, it’s just not given back to the system. In this case you should restart Sonic. This only occurs at import with a certain import speed that’s high.

[valeriansaliou]

On the total data size, yes, you should use regular filesystem commands to check that. Sonic doesn’t provide anything to get the total size.

[amirEBD]

Thanks for your answering Valerain, I have another question about a observation I had with sonic hard disk usage. As I create the docker container for sonic initially data/store directory is given but as I observed repeatedly the disk usage on / will increase realy much than expectation (about 100GBs). Will sonic use volumes other than data/store?

[valeriansaliou]

Sonic only uses this directory to store its data. You can expect everything to be stored in this single folder. It should contain a kv and a fst folder.

[Kryvchun]

Try to check how many clients are connected during ingesting.

telnet ::1 1491
START control <PASSWORD>
INFO
QUIT

It will print something like this:

RESULT uptime(33270) clients_connected(35) commands_total(437461) command_latency_best(1) command_latency_worst(490) kv_open_count(0) fst_open_count(0) fst_consolidate_count(0)