Enhance S3TransferManager S3(Native)ClientConfiguration to include additional CRT S3ClientOptions

[ron1]

Describe the feature

The S3TransferManager components S3ClientConfiguration, S3NativeClientConfiguration, and others are enhanced to allow customization of the following software.amazon.awssdk.crt.s3.S3ClientOptions properties:

String endpoint;
TlsContext tlsContext;

Is your Feature Request related to a problem?

I am unable to upgrade the aws sdk used in my application from sdk v1 to sdk v2 because the sdk v2 S3TransferManager does not allow the S3 endpoint nor the SSLSocketFactory to be customized.

Proposed Solution

Enhance several S3TransferManager components to allow customization of the CRT S3ClientOptions specified above.

Describe alternatives you've considered

No response

Acknowledge

• [x] I may be able to implement this feature request

Would you accept a PR that implements this feature?

AWS Java SDK version used

2.17.102

JDK version used

OpenJDK Runtime Environment (build 1.8.0_302-b08)

Operating System and version

CentOS 7.9

[debora-ito]

As @zoewangg mentioned in #2946, the V2 TransferManager is being refactored and the way to configure the S3 client will probably change - the plan is to allow to pass a S3Client to TransferManager, with all the client customizations you need.

Let's keep this open to track if the feature request is fulfilled after the refactor.

[ron1]

@debora-ito @zoewangg Now that v0.15.20 of the AWS CRT for Java has been released with the ability to Perform S3 get and put operations over HTTP as well as HTTPS, and with custom ports, would you reconsider accepting a S3TransferManager PR that adds just the endpoint (and not the TLSContext) as an additional S3ClientOption? The latest set of S3TransferManager changes made by @zoewangg should simplify the PR. Removal of the TLSContext from the PR should also reduce the scope of the PR.

We have an immediate need to use the S3TransferManager to perform get and put operations over HTTP with custom ports. Without this PR, we must copy, paste, tweak, and maintain more than six S3TransferManger classes since most of these classes are internal and therefore not reusable. Thanks for your consideration.

[zoewangg]

Hi @ron1, I think just exposing endpointOverride(URI endpointOverride); on the S3TransferManagerConfiguration is fine, but we still intend to get rid of the S3TransferManagerConfiguration in favor of S3AsyncClient in the near future, which means part of the code would be removed eventually before GA. If you are fine with it, we can merge the change.

[ron1]

Hi @zoewangg. I am fine with an interim solution that includes throw-away code. I will go to work on a PR that exposes endpointOverride(URI endpointOverride); Thanks much.

[ron1]

@zoewangg BTW, when you mentioned S3TransferManagerConfiguration in your comment, were you referring specifically to class S3ClientConfiguration, classS3TransferManagerOverrideConfiguration, or some other class?

[zoewangg]

Apologies for the delayed response, I meant to say S3ClientConfiguration .

[dvisigalli]

Hi @ron1, I think just exposing endpointOverride(URI endpointOverride); on the S3TransferManagerConfiguration is fine, but we still intend to get rid of the S3TransferManagerConfiguration in favor of S3AsyncClient in the near future, which means part of the code would be removed eventually before GA. If you are fine with it, we can merge the change.

@zoewangg with this approach will also be possible to set a specific proxy configuration to S3 Transfer Manager? Is there already a way to do this? At the moment i'm using an AwsCrtAsyncHttpClient instance configured on a S3AsyncClient and it seems to work behind a proxy (i can connect to buckets and list the objects). Unfortunately i still have issues with S3 Transfer Manager, which fails in downloading objects (AWS_ERROR_INVALID_ARGUMENT(34) when i override the endpoint, CRT error code: 1069 not overriding endpoint). Could this be a caused by Transfer Manager that is simply not using the correct proxy?