objstore
objstore copied to clipboard
thanos-io
s3: add DisableMultipart option
- [X] I added CHANGELOG entry for this change.
- [ ] Change is not relevant to the end user.
Changes
- s3: add DisableMultipart option to force disabling multipart all together.
Verification
Checked that the uploaded object was not upload with multipart
I'd like a review if that's possible please :) thx
@yeya24 @saswatamcode @brancz
Hi @fatpat, thanks for the contribution. May I know the usecase for disabling it? And maybe it is worth checking the conflicts since I just merged another pr.
Hi @fatpat, thanks for the contribution. May I know the usecase for disabling it? And maybe it is worth checking the conflicts since I just merged another pr.
Hello @yeya24,
I just rebase.
The motivation comes from a need in mimir that use objstore as a dependency. For the motivation I'll copy the justifcation I wrote on the mimir PR 7350:
we discovered that we could achieve better GET performances from our internal objecstorage with multipart disabled.
our use case is a high load with a high frequency (1s interval). We were trying to play with the part size while we discovered it was not possible do disable it all together (thanos-io/objstore resets part size if the object size is higher than the part size).
Moreover all S3 clients have an option to disable multi-part uploads. Not having the option in mimir looks like something was missing.
I am waiting to see if I can get another pair of eyes on this... Maybe @fpetkovski @kakkoyun @saswatamcode?
Sorry I forgot to merge it. Thanks for the reminder.
do you want me to rebase the PR ?
I am not sure why this pr didn't get auto merged. Maybe rebase will help
done