High memory usage and page faults on init and load

[aristaeus]

While experimenting with rdedup on my RAM limited VPS I noticed that the init and load commands used around 1GB of memory. This caused the program to swap which made it rather slow. rdedup store does not suffer from the issue and is basically instant.

I ran /usr/bin/env time -v rdedup -vvv init on my desktop. -vvv didn't seem to reveal anything enlightening, but time showed that the program had a lot of page faults:

Maximum resident set size (kbytes): 1054192
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 262849

I'm running rdedup 3.1.1, and both machines are on NixOS 19.03.

[dpc]

Page faults are probably just lazy memory allocation. It's just a symptom of memory pressure.

rdedup load is much less optimized then store (since in my use case, it's a much less common operation, I never got into optimizing it).

If you or anyone is interested in making this better, I am happy to help and answer questions, but I probably won't find time myself to it in near term.

[aristaeus]

No worries. As you said - it's a low frequency operation so it's not a major issue. Maybe I'll have a look at optimising it, though I'm not a rust dude, so it won't be any time soon.