objstore icon indicating copy to clipboard operation
objstore copied to clipboard

WIP: Allow prefix listing without directory delimiter

Open simonswine opened this issue 2 years ago • 0 comments

This change allows to list buckets by prefix with the parameter IterParams.WithoutAppendDirDelim is set, without appending a / in the end.

This allows to list huge "directories" bit by bit and in parallel. It is quite useful to list a huge amount of ULID (or other lexicographical ordered keys).

This would allow to preselect a creation time of blocks by e.g. listing /01H19* ULIDs, we would only list blocks created between 2023-05-25T08:21:51.744Z and 2023-05-25T17:41:06.175Z.

Before I spend more time on this, I wonder how people are feeling about it and if this useful for others.

Outstanding tasks:

  • [ ] Implement this for filesystem
  • [ ] Test coverage
  • [ ] Implement for other providers than azure/gcs/s3

simonswine avatar May 25 '23 10:05 simonswine