Tracking issues of RFC-3911: Deleter API

[Xuanwo]

• RFC: https://github.com/apache/incubator-opendal/pull/3911

Tasks

• [x] https://github.com/apache/opendal/pull/5392
  • Raw API: Add oio::Delete
  • Raw API: Change delete to return oio::Delete in Accessor
  • Raw API: Remove batch API
  • Public API: Add deleter to Operator
• [x] Public API: Deprecate remove, remove_via APIs
• [x] Remove batch capability
• [ ] Add recursive support
  • [x] https://github.com/apache/opendal/pull/6824
  • [ ] https://github.com/apache/opendal/pull/6827
  • [ ] azdls
  • [ ] webdav
  • [ ] webhdfs
  • [ ] hdfs
  • [ ] hdfs_native
  • [ ] dropbox
• [ ] Add concurrent support

In the future

• [ ] Implement dtrace support for lister and deleter
• [ ] Implement oteltrace support for deleter
• [ ] implement batch delete for obs
• [ ] implement batch delete versioned object for oss
• [ ] Implement delete version check for deleter
• [ ] Add recursive support for azdls

[howiieyu]

If I understand correctly, there are a few typos in the RFC.

L37 in RFC text.

+  pub async fn delete_with(&self, path: &str) -> FutureDelete;

delete_with already return Future, async keyword is not necessary.

L40 in RFC text.

+  pub async fn deleter_with(&self) -> FutureDeleter;

same as above.

L99 in RFC text.

let deleter = op.deleter()
  // Allow up to 8 concurrent delete tasks, default to 1.
  .concurrent(8)
  // Configure the buffer size to 1000, default value provided by services.
  .buffer(1000)
  .await?;

// Add a single file to the deleter.
deleter.delete(path).await?;

// Add a stream of files to the deleter.
deleter.delete_all(&mut lister).await?;

// Close deleter, make sure all input files are deleted.
deleter.close().await?;

op.deleter() should be op.deleter_with()

[Xuanwo]

Thanks a lot! Would you like to submit a PR to fix them all?

[meteorgan]

Hi, @Xuanwo Have you started working on this issue ? I'm interested in it.

[Xuanwo]

Hi, @Xuanwo Have you started working on this issue ? I'm interested in it.

I'm currently working on this and will create separate tasks based on my progress.

[rich7420]

Hello , could I try this issue ? It seems that there still have some files in core and bindings needs to do. cc. @Xuanwo , @tisonkun

[tisonkun]

@rich7420 Sure. Go ahead and feel free to ping me as a reviewer for your PR.

[rich7420]

thanks @tisonkun !

[Xuanwo]

We have two blockers for this issue. One is that we don’t yet support recursive. we’ve made some progress in https://github.com/apache/opendal/pull/5762, but it’s not clear how to move forward. The other is that we don’t yet support concurrent.

Other issues include implementing batch delete for OBS and implementing batch delete for versioned objects on OSS. We also need to check all services that support batch delete.

[meteorgan]

Eight months have passed and a lot has changed, I will take some time to revise #5762 later. Maybe we can just pick one solution for now, even if it's not perfect.

[Xuanwo]

Eight months have passed and a lot has changed, I will take some time to revise #5762 later. Maybe we can just pick one solution for now, even if it's not perfect.

Oh, sorry for missing these comments. I’ve been working on this and have added https://github.com/apache/opendal/pull/6824, maybe you’ll be interested to take a look.

[meteorgan]

Do we need a feature to delete all versions of a file or an entire path ?

[Xuanwo]

Do we need a feature to delete all versions of a file or an entire path ?

Already supported by:

let file_versions = op.lister_with(path).versions(true).await?;
op.delete_try_stream(file_versions).await?