Document vm_overcommit requirement (was Seemingly High Quanity of Memory Requested)

[hannah-leitheiser]

Docker Message:

moor-daemon | 2024-02-18T09:28:15.014657Z INFO main moor_daemon: crates/daemon/src/main.rs:185: Daemon starting... moor-daemon | The application panicked (crashed). moor-daemon | Message: Unable to create pager: InitializationError("Mmap failed for size class block_size: 4096, virt_size 1099511627776: Cannot allocate memory (os error 12)")

crates/db/src/lib.rs:64 RelBoxWorldState::open(self.path.clone(), self.memory_size.unwrap_or(1 << 40));

I don't know Rust, but this program appears to request a terabyte of memory, which is not available on my system. I apologize if there is simply a setting I neglected.

[rdaum]

Yes, it requests that memory but only uses what it needs, part of the DB pool logic. The OS properly configured should still give it. It may be that vm.overcommit needs to be turned on. I'll have to check that and document it.

[rdaum]

You may need to set /proc/sys/vm/overcommit_memory to "1".

https://www.baeldung.com/linux/overcommit-modes

[rdaum]

For more explication.

Moor builds out its own custom database and does not use an off the shelf key value store. In the future I may go back to providing the option of an alternate backend based around something like RocksDB or LLMDB, etc. But for now in order to make the transaction mgmt side to moor work performantly and be fully under my control, this is how it is.

The buffer pooling for that DB uses strategies from the LeanStore and, in particular, Umbra papers: https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf

In that system large quantities of virtual memory are requested, in multiple page sizes. But these are virtual memory pages, and are not actually resident process allocated. In reality the OS only actually really allocates pages from that pool as the pages are used. You can see the actual memory use under the process RSS in top etc rather than the virtual memory it has requested. This technique is used to efficiently allocate and free pages of varying size for database usage.

But it does require that the overcommit_memory option in the kernel be turned on.

Keep in mind this is all code under heavy development, no bug-free or production-quality guarantees are made.

[hannah-leitheiser]

Hello, rdaum. Thank you for your work and feedback. I did a fresh install of Docker on Ubuntu 22.04 for your project.

The OS properly configured should still give it.

But I also have a KVM server running, so maybe that's interfering and my situation represents a special case.