opendal icon indicating copy to clipboard operation
opendal copied to clipboard

Add native support for monoio project

Open Xuanwo opened this issue 1 year ago • 4 comments

Apply for this project: https://summer-ospp.ac.cn/org/prodetail/241110346

Goal

Add monoio based fs support as mentioned in https://github.com/apache/opendal/issues/4520.

The usage will look like:

let op = Operator::via_map(Scheme::MonoioFs, map)?;
let bs = op.read("path/to/file").await?;

Requirement

  • Know about monoio
  • Know about how to build/test rust project
  • Better to have some knowledge about io-uring
  • The current fs API in monoio is incomplete; we may need to collaborate with upstream to finalize them.

Mentorship

Mentor: @Xuanwo Discussion: mailing list and discord

Xuanwo avatar Apr 30 '24 09:04 Xuanwo

@Xuanwo I'm interested in this.

hoslo avatar May 10 '24 06:05 hoslo

@Xuanwo I'm interested in this.

Thank you for your interest. However, this project is designated as an OSPP project; perhaps you would like to participate as a mentor instead?

Xuanwo avatar May 10 '24 07:05 Xuanwo

@Xuanwo I'm interested in this.

Thank you for your interest. However, this project is designated as an OSPP project; perhaps you would like to participate as a mentor instead?

I just noticed this, you are better as a mentor, I can go and solve other issues.

hoslo avatar May 10 '24 08:05 hoslo

Task list to track progress of monoiofs

  • [x] init monoiofs service, introduce feature flag and dep (#4855)
  • [x] monoio runtime wrapper, thread management and operation dispatch (#4885)
  • [x] open, read, write, stat & delete (#4944)
  • [x] behavior tests (#4944)
  • [x] append, create_dir, rename & copy (#5041)
  • [ ] benchmarks (#5095)

Blocked by monoio

  • [ ] vectored io (bytedance/monoio#266)

NKID00 avatar Jun 26 '24 07:06 NKID00

Hi, @NKID00, are you interested in getting some benchmark data about fs and monoiofs?

Xuanwo avatar Mar 12 '25 13:03 Xuanwo

Hi, @NKID00, are you interested in getting some benchmark data about fs and monoiofs?

I've measured some data while writing the final report of the OSPP project. You can find the raw data in #5095 and there was a chart in the report. I can repost it here:

Image

These data may be a bit out dated, though.

NKID00 avatar Mar 12 '25 13:03 NKID00

Cool!

Xuanwo avatar Mar 12 '25 13:03 Xuanwo

Most issues have been addressed. Let's close this one.

Xuanwo avatar Mar 18 '25 07:03 Xuanwo