lsm-tree icon indicating copy to clipboard operation
lsm-tree copied to clipboard

memtable/skiplist: add a purpose-built skiplist

Open ajwerner opened this issue 6 months ago • 1 comments

Fixes https://github.com/fjall-rs/lsm-tree/issues/95.

Supersedes #98

ajwerner avatar May 18 '25 04:05 ajwerner

With the Borrow<Q> trait bound it's not possible to use InternalKeyRef that would allow to not build an InternalKey (which involves a heap allocation). Need to use equivalent::Comparable instead.

See: https://github.com/fjall-rs/lsm-tree/pull/98

marvin-j97 avatar May 18 '25 15:05 marvin-j97

Todo:

  • Evaluate memory overhead vs crossbeam skiplist
  • Evaluate read and write latency vs crossbeam skiplist
  • Try in a full system (heavily cached) benchmark (https://github.com/marvin-j97/rust-storage-bench) vs crossbeam skiplist

marvin-j97 avatar Jul 12 '25 00:07 marvin-j97

I've tried rebasing this onto the v3 branch.

Outlook now: we don't need a free list because all insertions will be unique (because of unique sequence number).

marvin-j97 avatar Sep 06 '25 20:09 marvin-j97

Removing the const generic N parameter is actually a bit harder than expected because it defines the layout of Buffer<N>; changing it to static makes pointer derefs fail with misaligned pointer dereference.

marvin-j97 avatar Sep 08 '25 16:09 marvin-j97

Makes sense. One day if I find time I'll finish up some of the ideas here.

ajwerner avatar Nov 02 '25 20:11 ajwerner

I deleted the 3.0.0 branch, so this needs to rebased on main now

marvin-j97 avatar Nov 02 '25 20:11 marvin-j97

@ajwerner Can you reopen this PR against main?

marvin-j97 avatar Nov 07 '25 13:11 marvin-j97