horaedb
horaedb copied to clipboard
Tracking issue about todo in distributed wal
Description There are some enhancements in wal needing to do in the futures, just mark them now.
Proposal
On message
- [x] Now manifest based on wal need to scan twice from storage while doing snapshot. Add a cache for this may be better.
- [ ] In wal's message queue implementation, we now sync the snapshot to remote in every flushing. Indeed, we should just need to do the sync work while flushing after dropping table.
- [x] Now, the desigin of wal on message queue is based on the thread model of
one writer for one table
, maybe we should remove this assumption. - [x] Define some iterators with more explicit semantics.
- [ ] Now we init region(heavy work with network IO) even if it is just created, we should change
MessageQueue
to know whether it is a new region. - [x] We should disable the ttl of topic in kafka, but it is not supported in
rskafka
now. It is important so I plan to use another crate to finish it temporarily.
On OBKV
- [ ] Rename the table name prefix.
- [x] Change the meta table's sharding key to
table id
.
Manifest(on wal)
- [x] Use the table id as region id.
Additional context