opendal icon indicating copy to clipboard operation
opendal copied to clipboard
verified publisher icon apache

new feature: Options API for `object_store`

Open linhr opened this issue 1 year ago • 1 comments

Feature Description

The object_store::ObjectStore trait has APIs such as put_opts(), put_multipart_opts() and get_opts(). It would be desirable to implement these APIs in the OpenDAL integration.

Problem and Solution

In the OpenDAL integration, these APIs return an "unimplemented" error, since #3713. I guess it might be possible to process the object_store options in the arguments and convert them to OpenDAL operations. I'm not sure if all the options can be supported though.

Additional Context

I'm using Apache DataFusion to build query engines. DataFusion uses the object_store::ObjectStore trait and its options API, and I'd like to adapt it to OpenDAL so that more data sources can be integrated.

Are you willing to contribute to the development of this feature?

  • [ ] Yes, I am willing to contribute to the development of this feature.

linhr avatar Sep 12 '24 06:09 linhr

Thank you very much for completing this. I will look into the API to see how we can implement it.

Xuanwo avatar Sep 12 '24 07:09 Xuanwo

Hi, @linhr. Thanks to @meteorgan's work, I beleive this integration should just work: https://github.com/apache/opendal/pull/5543

Xuanwo avatar Jan 14 '25 04:01 Xuanwo

This is an exciting progress! Thanks for having this!

I'll leave this issue open to track the implementation for put_multipart_opts, but it seems the usage of OpenDAL with DataFusion is unblocked now!

linhr avatar Jan 14 '25 08:01 linhr

Hi, @meteorgan can we close this issue now?

Xuanwo avatar Mar 11 '25 08:03 Xuanwo

Hi, @meteorgan can we close this issue now?

we haven't implemented put_multipart_opts yet. I'm going to complete it in the next few weeks. and we need to finish #4075 first.

meteorgan avatar Mar 11 '25 09:03 meteorgan

and we need to finish #4075 first.

I'm not sure if it's a good idea to implement tags support. Since it's not widely used, let's implement put_multipart_opts directly without it.

Xuanwo avatar Mar 11 '25 10:03 Xuanwo

I'm not sure if it's a good idea to implement tags support. Since it's not widely used, let's implement put_multipart_opts directly without it.

The opts in put_multipart_opts include tags and attributes. It seems both of them are not widely used ?

meteorgan avatar Mar 11 '25 12:03 meteorgan

No action required at this time, so I’m closing this for now. Please feel free to reopen it if you believe further work is needed.

Xuanwo avatar Apr 20 '25 05:04 Xuanwo