horaedb icon indicating copy to clipboard operation
horaedb copied to clipboard

Tracking issue about todo in distributed wal

Open Rachelint opened this issue 2 years ago • 0 comments

Description There are some enhancements in wal needing to do in the futures, just mark them now.

Proposal

On message

  • [x] Now manifest based on wal need to scan twice from storage while doing snapshot. Add a cache for this may be better.
  • [ ] In wal's message queue implementation, we now sync the snapshot to remote in every flushing. Indeed, we should just need to do the sync work while flushing after dropping table.
  • [x] Now, the desigin of wal on message queue is based on the thread model of one writer for one table, maybe we should remove this assumption.
  • [x] Define some iterators with more explicit semantics.
  • [ ] Now we init region(heavy work with network IO) even if it is just created, we should change MessageQueue to know whether it is a new region.
  • [x] We should disable the ttl of topic in kafka, but it is not supported in rskafka now. It is important so I plan to use another crate to finish it temporarily.

On OBKV

  • [ ] Rename the table name prefix.
  • [x] Change the meta table's sharding key to table id.

Manifest(on wal)

  • [x] Use the table id as region id.

Additional context

Rachelint avatar Nov 03 '22 08:11 Rachelint