leofs icon indicating copy to clipboard operation
leofs copied to clipboard

[leo_storage] Unbounded prefix_search could block leo_storage for long time

Open windkit opened this issue 7 years ago • 0 comments

Description

leo_object_storage_api:fetch_by_key is unbounded, for a bucket / sub-directories with tons of objects, it would block the metadata server.

Solution

Pass the MaxKeys down the pipe to limit the listing process.

Currently sub-results from metadata servers are only concatenated but not sorted, cropped the list at MaxKeys would return incorrect paged result. (As in https://github.com/leo-project/leofs/issues/821)

Metadata1: obj0, obj13 <- sorted
Metddata2: obj1, obj2 <- sorted

Combined List: [obj0, obj13, obj1, ojb2] <- not sorted
Cropped List: [obj0, obj13] <- not sorted
Correct List: [obj0, obj1]

The cropped list is incorrect and therefore the next marker is incorrect

To solve the issue, we need to merge / concat + sort the sub-results

Related Issue

https://github.com/leo-project/leofs/issues/821

windkit avatar Sep 08 '17 03:09 windkit