liballocs
liballocs copied to clipboard
stephenrkell
liballocs should be meta-complete
At present, liballocs has blind spots about its own abstractions -- specifically, about its own allocations and about the "uniqtype of uniqtypes".
It would be good to eliminate these. There is a rhetorical motivation for doing so -- showing we can describe our own stuff is a good design-level sanity check, making the design more persuasive. But also, being able to call on some of our own functions when managing allocations might make our code a bit simpler and more uniform, such as in how we manage and free bigallocs' metadata allocations.
This would mean (1) liballocs's private malloc being reified as an allocator, (2) ditto for any other allocations we do, e.g. memtable mappings, and (3) a working/precise uniqtype of uniqtypes, including a make_precise function. There maybe other aspects to meta-completeness that I'm overlooking right now.
Another issue is the allocsite metadata, including static allocation info, for the meta-objects. I think this would need to be generated online, rather than precomputed in a -meta-meta .so.
It might be reasonable to make bigallocs for our memtables and any other huge allocations, even though we obviously don't want to hook them into the pageindex. The pageindex itself is such a structure.
More generally, we might want a story on 'sparse' bigallocs for which the pageindex isn't good. Maybe a slow path: if the pageindex fails, consult a side structure that just records the sparse stuff.
If the sparse stuff is type-uniform (any memtable), it's trivial to record its type info in its bigalloc record.
Our own __private_malloc interface could be extended to do the hooky stuff. This will be easier when we have separated out distinct mallocs (see #21).
Anything we mmap ourselves is also in scope here. It should fall under some bigalloc. Should we hook our own mallocs (just differently from how we hook user mallocs)?
This affects e.g. how we mmap the section headers and static symtab of loaded ELF files.
The pageindex itself should have a bigalloc. But then we'd have to memset a region of the pageindex. The full pageindex for a 47-bit address space and 12-bit pages has 2^35 entries, each 2 bytes, so 64GB in size. How much of the pageindex does it take to cover 64GB? It's 2^24 pages so 2^25 bytes. That's too much to allocate eagerly. So maybe we need to use /dev/ones if we want to be meta-complete. We could define the pageindex as being bigalloc MAX. Some kind of masking may be necessary, since currently max-bigalloc is 1023 rather than 65535 (all-ones).