raft-rs icon indicating copy to clipboard operation
raft-rs copied to clipboard

Support to fetch entries asynchronously

Open hicqu opened this issue 3 years ago • 4 comments

Is your feature request related to a problem? Please describe.

In the current API new appended entries and committed entries are fetched into Ready synchronously. However In many cases we want to fetch them asynchronously. For example:

  • There are lots of entries need to be fetched and it is not expected to block the current thread;
  • The memory usage is almost reach the limit, it it not expected to fetch lots of entries into memory.

Describe the solution you'd like

Adjust Ready to

fn Ready::entries(&self) -> Either<&[Entries], Range<u64>>;
fn Ready::committed_entries(&self) -> Either<&[Entries], Range<u64>>;

And add 2 new fields into Config:

pub struct Config {
    fetch_entries_asynchronously: bool,
    fetch_committed_entries_asynchronously: bool,
}

hicqu avatar Jun 28 '21 08:06 hicqu

@BusyJay @gengliqi @Little-Wallace PTAL.

hicqu avatar Jun 28 '21 09:06 hicqu

I think we should also consider the entries in AppendEntries msg. They can be fetched asynchronously too.

gengliqi avatar Jun 28 '21 09:06 gengliqi

Log term should also be checked.

BusyJay avatar Jun 28 '21 10:06 BusyJay

Log term should also be checked.

How about maintain terms with a Vec<(term, index) in memory for uncompacted entries? And only the first entry in a term needs to be recorded, so it won't be expensive.

hicqu avatar Jun 28 '21 10:06 hicqu