opendal icon indicating copy to clipboard operation
opendal copied to clipboard

feat(adapter/kv): support async iterating on scan results

Open PragmaTwice opened this issue 1 year ago • 1 comments

Which issue does this PR close?

Closes #3639.

Rationale for this change

Instead of performing a full scanning and returning a Vec<String>, now we introduce an associated type into kv::Adapter:

type Scanner: Scan;

So that the implementation can do paged scanning and returns a ScanIter, and users can iterate over it to get keys in batches from the server.

What changes are included in this PR?

  • add associated type ScanIter into kv::Adapter and refactor KvLister and Adaptor's Accessor impl;
  • refactor all services using kv::Adapter, especially for which has list cap;
  • for sqlite (via sqlx): now, instead of using fetch_all, we can use fetch to fetch data in batches instead of fetching all data.

Are there any user-facing changes?

No changes for end users.

PragmaTwice avatar Oct 20 '24 08:10 PragmaTwice

There are some critical bug fixes that need to be released. I will merge this after we create a new release.

Xuanwo avatar Oct 24 '24 11:10 Xuanwo