client-rust icon indicating copy to clipboard operation
client-rust copied to clipboard

Add new stream scan APIs for transaction

Open yongman opened this issue 3 years ago • 1 comments

Add new scan_stream, scan_keys_stream, scan_reverse_stream, scan_keys_reverse_stream APIs for transaction scan. Use these APIs, scan request will be invoke request to one region at a time in a configurable batch size, which is more reasonable when user scan range is wide-range, such as all key space.

yongman avatar Sep 07 '22 04:09 yongman

@ekexium @iosmanthus PTAL

yongman avatar Sep 07 '22 06:09 yongman

IIRC, you can limit the scan size in batch using the current API like:

let mut start_key = Key::from(vec![]);
let end_key = Key::from(vec![]);
let limit = 100;

let mut buffer = vec![];
let mut last_len = 0;

loop {
    buffer.extend(client.scan((start_key, end_key), limit).await?);
    if buffer.len() - last_len < limit {
        break;
    }
    last_len = buffer.len();
    start_key = buffer.last().unwrap().0.clone();
    start_key.push_zero();
}

I agree with you that an iterative api is more ergonomic, but I think we can make it a wrap function like above instead of internal mechinism like in this PR.

andylokandy avatar Jul 09 '23 19:07 andylokandy