liburkel icon indicating copy to clipboard operation
liburkel copied to clipboard

Make `mmap` in posix_io more configurable?

Open nodech opened this issue 1 year ago • 1 comments

In different environments where it's necessary to not use a lot of memory, I think it should be possible to disable mmap altogether or have it behind flag that is controlled on tree creation.

Mapped files (given that current configuration is 2GB) can take a lot of memory - 1 for working file (which is constantly remapped(unmapped and mapped with new size) on each write) and others if there's need to look up something from the tree. For each open of the past file (e.g. 2 entries from 2 files) can cause extra 2GB file being mapped causing RSS to skyrocket. (I saw 24GB being used for hsd using nurkel - disabled mapping altogether locally and issue went away)

Maybe don't use mmap for readonly past files ? I think having MAX 2GB in the memory (if performance boost is worth it) is OK, but frequent past file look ups - nope.

So basically, either remove URKEL_O_MMAP from READ_FLAGS or make them somehow part of the API?

nodech avatar Feb 24 '23 13:02 nodech