wickdb
wickdb copied to clipboard
Fullstop000
Feature: support concurrently insert batches into memtable
In the origin version of leveldb, the write path is like below:
write
write + write
+ | write +
| | + |
| | | |
| | | |
+---v-------v-----v-----v---+
| |
| Mutex<queue> |
| |
+-------------+-------------+
|
|
| group batch
| <single thread>
|
v
+--------+--------+
| |
| Memtable |
| |
+-----------------+
Writing batches concurrently will result in a single thread ingestion into memtable and then notify all the relate blocked write request threads. Maybe we can support concurrently inserting batches into memtable , which should improve the write performance a lot.
This feature includes following tasks:
- [x] Introduce an thread-safe concurrent skiplit (https://github.com/Fullstop000/wickdb/pull/85)
- [ ] Make the current queue scale with inserting threads
The current Skiplist approach doesn't support concurrent inserting. Maybe we should port InlineSkipList from rocksdb https://github.com/facebook/rocksdb/blob/master/memtable/inlineskiplist.h
The current Skiplist approach doesn't support concurrent inserting. Maybe we should port
InlineSkipListfrom rocksdb https://github.com/facebook/rocksdb/blob/master/memtable/inlineskiplist.h
I would like to try on this.
@accelsao Also you may refer to https://github.com/dgraph-io/badger/blob/master/skl/skl.go, which looks like a more clean version of the rocksdb's
badger arena is different from rocksbdb, do you know which one is better to write in rust? badger save node offset instead of a pointer.
badger arena is different from rocksbdb, do you know which one is better to write in rust? badger save node offset instead of a pointer.
I think they serve the same purpose with a different approach. Using the current arena should be enough.
How do you think about crossbeam/skiplist, is it a good way to achieve concurrent insert for memtable?
How do you think about crossbeam/skiplist, is it a good way to achieve concurrent insert for memtable?
It may need some extra work to fit the Comparator
@accelsao I've tested the offset-based Arena and it seems more suitable with your inline skiplist implementation.
Do you have time to bring it to your current PR? Maybe I can help :)