poolpOrg
excessive memory usage
User reported excessive memory usage when performing snapshot of large number of files and directories, exhausting his 16GB RAM on an 8 core machine.
Using a relatively similar setup, 8 core and 16GB of RAM, I could not reproduce but my test still consumed 1GB of RAM which seemed excessive.
I thought it could be caused by disk performances causing high parallelisation to buffer too much chunks for disk to keep up. User reported write performances 2x slower than mine which was another hint in that direction.
I tested by setting up a remote plakar repository and also a plakar repository on a SATA disk which both proved to consume more memory than when snapshotting to my SSD.
After a lot of experiments, I ruled out that this is just a buffering issue: reducing parallelisation and adding disk write semaphores does not help whatsoever.
The high memory consumption happens before the snapshot begins, when the source directories/files are being scanned and plakar's VFS layer creates its filesystem view.
This is confirmed by plakar scan /origin consuming excessive memory while it only does the filesystem traversal and tree build. I also ruled out that the scan itself is the issue by commenting the tree build.
The VFS structure needs to be redesigned and optimized...
It turns out the VFS data structure is too expensive.
I'm currently testing a custom trie
I have solved the issue in a PoC:
Long story short, regardless of the data size of a snapshot, the VFS space requirement grows with the number of files and directories within a snapshot. It doesn't matter that I use a mapping or a trie, if I have n million files in a snapshot that's n millions pathnames + n millions FileInfo structures to keep in memory, not counting the need to keep track of children per nodes which is possibly many millions additional entries.
The quick solution is to offload VFS to a disk-based database. I tested boltdb, sqlite and leveldb, the latter proved to be the most efficient both in disk write performance and in overall size with 1 million objects producing a 30MB database (scaling linearly, so I can easily predict what 10 or 100 million objects will produce). boldtb and sqlite were far behind in all tests.
The trade off to rely on disk is acceptable IMO given that obtaining the same result with an in-memory structure is A LOT of work, probably more than plakar all combined so there's no way it'll be achievable in short timespan.
An implementation of VFS & Index disk offloading has been written and committed in branches new-vfs and new-index.
They are functional and are being tested a bit more seriously before they are merged in main.