apache
Support `Operator::remove_all` in huaweicloud OBS Parallel File System
Describe the bug
Huaweicloud OBS has concept of directory, which isn't part of S3 protocol.
This difference brings some issues when performing recursive deletion.
For example, if we are trying to invoke Operator::remove_all on a prefix, it will try to stat the prefix first. OBS will return a directory object which cannot be deleted(will cause DirectoryNotEmpty error) if there is still objects in the directory.
Steps to Reproduce
op.write_with("/path/to/obj", payload).await.unwrap();
op.remove_all("/").await.unwrap(); // this will return error
Expected Behavior
No error.
Additional Context
No response
Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
OpenDAL now relies on list_with_recursive to perform remove_all. If storage services support list_with_recursive, we treat them as flat storage services, where /path/ is considered a flat key rather than a directory, allowing it to be safely removed. If not, as with fs, we list directories one by one and ensure the directory is removed at the end.
However, in obs case, it applies dir concepts at the level of list recursion. This can happen if users create an obs bucket with x-obs-fs-file-interface: Enabled which they called it Parallel File System
I think we need some special handling or even a new service for it.
By the way, the performance criteria for the parallel file system can be quite different from those of OBS, so please make sure you truly want to use it.
I've removed the bug tag from this issue since it's more about supporting a new feature. There are no actions we can take at the S3 service level.