opendal icon indicating copy to clipboard operation
opendal copied to clipboard

core: Implement concurrent read for blocking read

Open Xuanwo opened this issue 1 year ago • 6 comments

We can generate different ranges and then use rayon to concurrently fetch them.

Xuanwo avatar Apr 23 '24 15:04 Xuanwo

Does this require an RFC or can it be done directly?

hoslo avatar Apr 26 '24 06:04 hoslo

Does this require an RFC or can it be done directly?

We can do it directly but I want to know about your design first.

Xuanwo avatar Apr 26 '24 06:04 Xuanwo

There are two scenarios:

  1. We have an Range, split it into concurrent intervals and read it using rayon
  2. No Range, read in parallel with concurrent using 1024 * 1024 * 4 until we reach the end

hoslo avatar Apr 26 '24 08:04 hoslo

Do you have more details? What's the API will look like? How do we share the concurrent features between BlockingReader, StdReader and StdIterator?

Xuanwo avatar Apr 26 '24 09:04 Xuanwo

We add a field OpReader into BlockingReader and use it concurrent read in method read_at and read_into.

hoslo avatar Apr 26 '24 10:04 hoslo

We add a field OpReader into BlockingReader and use it concurrent read in method read_at and read_into.

I'm not sure if we are aligned. Please feel free to give it a try.

Xuanwo avatar Apr 26 '24 10:04 Xuanwo

Closing as no user really want this.

Xuanwo avatar Sep 15 '25 08:09 Xuanwo