aws
Independent configuration of threshold for multipart upload and copy
Describe the feature
MultipartS3AsyncClient allows configuring the multipart threshold in bytes:
S3AsyncClient.builder()
.multipartEnabled(true)
.multipartConfiguration(c -> c.thresholdInBytes(...))
As far as I can tell, this threshold is used for both put object and copy object to determine when to perform the operation as a multi part operation.
Contrast this to the v1.x TransferManager that had very different default thresholds for the two operations: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/TransferManagerConfiguration.java#L36-L42
One practical implication of this is that in multipart copy operations, metadata is not copied between objects, and must be done manually, so using a low threshold for multipart operations may require manual metadata copying for copies that otherwise would not require this. Raising the threshold though may adversely affect upload operations.
Since these two operations have different performance characteristics, is it reasonable to allow configuration of the thresholds independently?
Use Case
We perform both uploads and bucket-to-bucket copies with the same MultipartS3AsyncClient. In general we can guarantee an upper bound on object size (which is less that the single-part copy requirement), and when performing bucket-to-bucket copy we require metadata to be copied as well as object content.
This worked as expected with the v1.x TransferManager, with its thresholds of 16MB and 5GB for put and copy respectively.
We'd like to be able to configure the multipart copy threshold to something like 5GB to avoid having to manually copy metadata, without having to evaluate the effect this may have on multi part upload performance.
Proposed Solution
No response
Other Information
No response
Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
AWS Java SDK version used
2.38.7
JDK version used
17
Operating System and version
MacOS 26.1
