apache
GSoC Ideas: ofs, OpenDAL File System via FUSE
Background
OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. ofs can expose OpenDAL power in fuse way that allow users to mount storage services locally.
Objectives
Implement ofs, allowing users to mount storage services locally for read and write.
Features
In Scope:
- Continuous reading
- Continuous writing
- Random reading
- List dir
- Stat file
Out Scope:
- Random Write
- Xattrs
- Permissions
Tasks
- Implement features that in scope
- Implement tests suite
Recommended Skills
- Familiar with Rust
- Familiar with basic ideas of file system and fuse
- Familiar with OpenDAL Rust Core
Mentor
Mailing List: [email protected] Mentor: junouyang, Apache OpenDAL PMC Member, [email protected] Please leave comments if you want to be a mentor
Cross posted at https://issues.apache.org/jira/browse/GSOC-255
Can I mentor this GSOC project? And we have done some work for it before, but I am not sure if this project for GSOC is reasonable.
I interested in.Whether we discuss it before, we do the linux fuse or fuse3 first, and then consider the S3 support? @oowl
we do the linux fuse or fuse3 first, and then consider the S3 support? @oowl
we do not need to implement all these fuse libraries, just choose one of them.
I interested in
Welcome, but we need to wait for the nomination process of GSOC.
Welcome, but we need to wait for the nomination process of Gsoc.
The website shows that the organizations are announced on February 21st.
Will
ofs(opendal-fs?) be a library or a CLI tool?
I expect to have both lib and cli, but that depends on the code owner's decision.
Will
ofs(opendal-fs?) be a library or a CLI tool?I expect to have both lib and cli, but that depends on the code owner's decision.
Ok, I think it is a good way to separate these concerns for modularity and would be also helpful to link against it, when other projects, want to pull in that code. Not affiliated with it in any way, but I find it good, how abscissa applies that when initializing a new project, it's a good way to quickly start. So just leaving it here.
I'm familiar with the fuse before. I would like to be one of the mentor for this idea
I think we should update the goals of this idea, the goals listed above seems be implemented in https://github.com/apache/opendal/pull/4179
I think we should update the goals of this idea, the goals listed above seems be implemented in https://github.com/apache/opendal/pull/4179
Agreed, I will update the goals.
It seems to be a local mapping of cloud storage. It looks like switching out Datenlord's metadata engine, etcd, for a local SQLite should do the trick.
Rather than the data layer stuff, I'm more intersted about some decisions regarding metadata
Based on the existing ofs code, it seems. This idea would require implementing a reader and writer to handle prefetch and concurrent asynchronous write-backs, while also managing a certain amount of memory usage. There are also some things to consider on the metadata level.
It looks like my previous page_cache repo might be of some help.
I want to support Windows by implementing CloudFilter for ofs, I will write proposal later