hyrise
hyrise copied to clipboard
hyrise
Set mlockall
I just ran into a situation where Hyrise was paged out. This leads to really weird performance results. We can fix it by calling
mlockall(MCL_CURRENT | MCL_FUTURE);
in Hyrise::Hyrise(). We should discuss what the expected behavior is. For VMs or personal machines, we might be fine with paging, especially during the generation of benchmark data. An alternative would be to only call mlockall once all data is generated. In that case, we should also call it in the server.
Note: This seems to blow up jemalloc somehow. For some reason, it causes it to never return memory and for the binary to crash. Might mean that we have to abstain from locking the pages. :/
Reference: https://man7.org/linux/man-pages/man2/mlockall.2.html
mlockall() locks all pages mapped into the address space of the calling process. This includes the pages of the code, data, and stack segment, as well as shared libraries, user space kernel data, shared memory, and memory-mapped files. All mapped pages are guaranteed to be resident in RAM when the call returns successfully; the pages are guaranteed to stay in RAM until later unlocked.
If MCL_FUTURE has been specified, then a later system call (e.g., mmap(2), sbrk(2), malloc(3)), may fail if it would cause the number of locked bytes to exceed the permitted maximum (see below). In the same circumstances, stack growth may likewise fail: the kernel will deny stack expansion and deliver a SIGSEGV signal to the process.
Looks a little dangerous to me. Therefore, I would like to see this as an optional flag only.
Which part of this do you consider dangerous? The whole idea is "Never spill to disk, rather crash (sigsegv)"?
This makes total sense for servers with hundreds of gigabytes of RAM. But, for all other machines, we will probably end up with failures across the board. Therefore, I would implement this behavior as an optional setting.
@mrks is this related to the issues that we have in https://github.com/hyrise/hyrise/pull/2350?
