automq icon indicating copy to clipboard operation
automq copied to clipboard

[Enhancement] WriteAheadLog with sequentially callback

Open superhx opened this issue 6 months ago • 4 comments

Problem

Currently, BlockWALService persists data blocks in parallel, responding directly to the upper layer with success as soon as any data block is persisted, even if the previous data block has not completed persistence. Although this method provides "better" write latency, it shifts the responsibility of ensuring sequentiality to the upper layer, making the upper layer logic more complex.

Expectation

Implement a SequetialBlockWALService:

  • Provide the semantics of sequentially callback (the underlying can still be concurrent writing);
  • Through optimization of locks and models, the throughput and latency can match(or even better than) that of the original BlockWALService on Aliyun ESSD PL1 20GB 120MiB/s 2800 IOPS with write concurrency 8/16/32/64;
  • The format of the WAL needs to remain consistent with the previous one, and support the recovery logic with data holes of BlockWALService so that it can be directly replaced in the future;

superhx avatar Aug 03 '24 10:08 superhx