Customizing mimalloc to enable store and restore of a memory heap from one process to another for same binary

[beyonddream]

Hi,

At work, we are looking into experimenting with mimalloc as a general purpose memory allocator and possible replacement for jemalloc (from initial benchmark, it is showing promise and we see around ~1.5GB lesser memory consumption on avg. for a long running process). In addition, we are working on a project to improve startup time of our application. Currently the app is deployed in multiple machines (same/similar machine characteristics) and they all process a bunch of files before beginning to take user requests. This takes a long time on each machine and we are looking to speed up this process by making just one machine (producer) process the files and have its processed memory dump to file and directly load the file backed memory in other machines (consumers) so they can avoid the too long startup time. There is a separate effort to write a custom allocator for this use case but we wondered whether mimalloc can be customized to support it and started working on the PoC.

My current approach:

On Producer machine-

1. Create a new heap via mi_heap_new() that returns huge chunk of OS memory at a fixed base address, that is then used to process these files . (This also requires modification to mmap routine to use MAP_FIXED since we want to ensure OS returns address at given fixed base address)
2. Enable this new heap as current heap.
3. Any subsequent malloc()/free() will come from this heap.
4. When producer is done, then dump the whole heap in a file and ship them to the consumers.

On Consumer machines-

1. Create a new heap via mi_heap_new() and load the memory backed by file at the same fixed base address as on producer machine. (Note: This is crucial so all the internal pointers will be relative to the fixed base address and doesn't need to be changed/patched.)
2. Enable this new heap as current heap.
3. From now on, any subsequent malloc()/free() can come from this heap but also the app can use the existing memory objects created on the producer machines. (In a way, the idea is to patch up the mimalloc internal states so that it continues to work as if it had created these memory objects).

First of all, I would like to get feedback on this approach and also want to know if there are any alternate approach based on existing mimalloc features/api that will make this implementation easier. Also, given that this feature might be useful for others, is this something the upstream maintainers interested in merging to mimalloc ?

Please let me know if you have any questions!