apache
Adopt OpenDAL for accessing storage layer
Is there an existing issue for this?
- [X] I have searched the existing issues
Description of the bug
OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. Our goal is to deliver a comprehensive solution for any languages, methods, integrations, and services.
ParadeDB uses Apache OpenDAL to integrate with object stores. If we support OpenDAL, all databases that use OpenDAL will be able to use Hudi
Steps To Reproduce
n/a
Expected behavior
n/a
Screenshots / Logs
No response
Software information
n/a
Additional context
No response
there's a crate that brings opendal integrations to object_store hudi-rs uses: https://crates.io/crates/object_store_opendal It may be worth considering this instead of replacing it with opendal completely?
@Xuanwo and I had a chat - we don't have to bridge the APIs as the current object_store footprint is still small within the codebase - the API migration won't be too much of changes. And we should make use of opendal's capabilities to support more storage systems natively.
Hi, @xushiyan and @alberttwong, I'm working on this now. One blocker here is opendal doesn't support construct from url yet, I will add this support in opendal first.
Tracked at https://github.com/apache/opendal/issues/5407
Waiting for https://github.com/apache/opendal/pull/5482 and a new release that includes it.