objstore
objstore copied to clipboard
WIP: Allow prefix listing without directory delimiter
This change allows to list buckets by prefix with the parameter IterParams.WithoutAppendDirDelim is set, without appending a / in the end.
This allows to list huge "directories" bit by bit and in parallel. It is quite useful to list a huge amount of ULID (or other lexicographical ordered keys).
This would allow to preselect a creation time of blocks by e.g. listing /01H19* ULIDs, we would only list blocks created between 2023-05-25T08:21:51.744Z and 2023-05-25T17:41:06.175Z.
Before I spend more time on this, I wonder how people are feeling about it and if this useful for others.
Outstanding tasks:
- [ ] Implement this for filesystem
- [ ] Test coverage
- [ ] Implement for other providers than azure/gcs/s3