kuzu icon indicating copy to clipboard operation
kuzu copied to clipboard

Track ColumnChunk allocations through the BufferManager

Open benjaminwinger opened this issue 7 months ago • 0 comments

Implements #3660 (also see #3698)

I've merged the MemoryAllocator and MemoryManager; it wasn't obvious the purpose of the distinction (though it's occurred to me that we could have separate allocators for stuff using malloc like the ColumnChunks and our own allocator). Mainly I just wanted to avoid having to store separate references to the MemoryManager everywhere. The MemoryManager seemed better suited than the memory allocator to pass around to initialize column chunk buffers, however there wasn't an way of getting a MemoryManager from an existing buffer (just a MemoryAllocator). While I could have changed that, merging MemoryManager and MemoryAllocator seemed simpler.

Note that it was necessary to raise the buffer pool size used for the tests; that should be able to be lowered again once spilling to disk is implemented as part of testing that behaviour.

Issues

I was having some issues with hanging instead of throwing exceptions when doing large rel copies with a restricted buffer pool size (looking at the behaviour when there is not sufficient memory). As far as I can tell, sometimes, with a large buffer pool of ~2000-3000 MB the hanging seems to be it continuously swapping out pages in the hash index. It may be that it will eventually throw an exception, but there sometimes is enough free memory for it to almost continuously try to do a small amount of work.

What's more concerning is that restricting the buffer pool size to the minimum 64MB on the same tests it will hang, and running it with backtraces will show a backtrace from a buffer manager exception (when looking up in the hash index; a couple of times it was the assert in freeUsedMemory), but that exception isn't stopping execution and it continues running with high CPU usage (possibly related to #3459).

I suspect that spilling to disk (#3698) will more or less solve these issues, as we generally shouldn't need to operate in such a memory restricted environment, and with spilling to disk ideally won't need to actually fail unless both disk and memory are full, but I'm concerned that we don't seem to be failing reliably in those circumstances.

benjaminwinger avatar Jul 03 '24 19:07 benjaminwinger