leofs
leofs copied to clipboard
[leo_storage] Unbounded prefix_search could block leo_storage for long time
Description
leo_object_storage_api:fetch_by_key
is unbounded, for a bucket / sub-directories with tons of objects, it would block the metadata server.
Solution
Pass the MaxKeys down the pipe to limit the listing process.
Currently sub-results from metadata servers are only concatenated but not sorted, cropped the list at MaxKeys would return incorrect paged result. (As in https://github.com/leo-project/leofs/issues/821)
Metadata1: obj0, obj13 <- sorted
Metddata2: obj1, obj2 <- sorted
Combined List: [obj0, obj13, obj1, ojb2] <- not sorted
Cropped List: [obj0, obj13] <- not sorted
Correct List: [obj0, obj1]
The cropped list is incorrect and therefore the next marker is incorrect
To solve the issue, we need to merge / concat + sort the sub-results
Related Issue
https://github.com/leo-project/leofs/issues/821