automq
automq copied to clipboard
[Enhancement] WriteAheadLog with sequentially callback
Problem
Currently, BlockWALService persists data blocks in parallel, responding directly to the upper layer with success as soon as any data block is persisted, even if the previous data block has not completed persistence. Although this method provides "better" write latency, it shifts the responsibility of ensuring sequentiality to the upper layer, making the upper layer logic more complex.
Expectation
Implement a SequetialBlockWALService
:
- Provide the semantics of sequentially callback (the underlying can still be concurrent writing);
- Through optimization of locks and models, the throughput and latency can match(or even better than) that of the original
BlockWALService
on Aliyun ESSD PL1 20GB 120MiB/s 2800 IOPS with write concurrency 8/16/32/64; - The format of the WAL needs to remain consistent with the previous one, and support the recovery logic with data holes of
BlockWALService
so that it can be directly replaced in the future;