S3TransferManager V2 - Support for cross region copy

[freshmanken]

Describe the feature

Use S3TransferManager v2 to copy object cross region

Use Case

I want to copy s3 objects through S3TransferManager from multiple regions to US-WEST-2. I saw S3 CLI / S3AsyncClient have a way to provide region for both source / destination bucket. However after some try out didn't find a way to do in S3TransferManager. S3ClientConfiguration only accept 1 region parameter, experimented with it but not luck.

Would like to see if we have an existing way to achieve this or this will be a feature request?

Proposed Solution

No response

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

AWS Java SDK version used

2.0

JDK version used

11

Operating System and version

Amazon Linux 2 - Lambda

[zoewangg]

Hi @freshmanken thanks for reaching out. It's not supported right now, marking this as a feature request

[debora-ito]

Same feature request, for S3CrtAsyncClient: https://github.com/aws/aws-sdk-java-v2/issues/4037.

[zoewangg]

We have added cross region copy support in 2.20.99. You just need to enable crossRegionAccessEnabled on the S3 CRT client builder.

        s3CrtAsyncClient = 
                 S3AsyncClient.crtBuilder()
                                        .crossRegionAccessEnabled(true)
                                        .build();

Closing the issue.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[VIJAY-SUBRAMANIAM]

Enabling cross-region copy support on the S3 CRT client builder doesn't seem to fix the issue whereas Copy object request works with NettyNioAsyncHttpClient. multi-part copy using crt client also working fine

Java version amazon-corretto-20.0.1.9.1-windows-x64-jdk s3/crt version 2.20.126 tm.copy(copyRequest);

S3AsyncClient .crtBuilder() .region(region) .crossRegionAccessEnabled(true) .credentialsProvider(credentials) .minimumPartSizeInBytes(MIN_PART_CONTENT_SIZE) .maxConcurrency(maxConcurrency) //ceil(targetThroughputInGbps/4) * maxConcurrency .build();

software.amazon.awssdk.core.exception.SdkClientException: Failed to retrieve metadata from the source object. Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: K58YMSYFGWAX9XN6, Extended Request ID: yEecoMoUxfoz0E1EHz2vM5YZ1a/yT/Ni2HvlxYOPUL0f3A6fDFjZ3YYrvtB/z94qKX3ZAWz2z6s=) at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.20.126.jar:?] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108) ~[aws-xml-protocol-2.20.126.jar:?] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85) ~[aws-xml-protocol-2.20.126.jar:?] at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43) ~[aws-xml-protocol-2.20.126.jar:?] at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270) ~[sdk-core-2.20.126.jar:?] at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89) ~[sdk-core-2.20.126.jar:?] at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?] at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132) ~[sdk-core-2.20.126.jar:?] at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275) ~[utils-2.20.126.jar:?] at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224) ~[utils-2.20.126.jar:?] at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157) ~[utils-2.20.126.jar:?] at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819) ~[?:?] at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?] at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2227) ~[?:?] at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:136) ~[s3-2.20.126.jar:?] at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:125) ~[s3-2.20.126.jar:?] at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:94) ~[s3-2.20.126.jar:?] at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24) ~[aws-crt-0.24.0.jar:0.24.0]