aws-sdk-java-v2
aws-sdk-java-v2 copied to clipboard
aws
Transfer Manager is not able to connect to generic S3 with http protocol
Describe the bug
Configuring S3TransferManager with a custom endpoint of the Minio solution makes it to fail to connect.
I'm using the S3ClientConfiguration.Builder endpointOverride() option, using an address like: "http://10.255.0.18:9000"
I can use the transferManager without any issue with AWS. I can also use regular S3Client from sdk-v2 with this local Minio server. It's only the transfer Manager who fails to connect with this exception:
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
Enabling debug log I can see that connection is refused because apparently, it is tried to open the socket using https and port 443, even if a little bit earlier in the same look it is possible to see that http was set up:
DEBUG] [2022-08-11T14:56:01Z] [00007f035f7fe700] [connection-manager] - id=0x7f03e90db220: Acquire connection
[DEBUG] [2022-08-11T14:56:01Z] [00007f035f7fe700] [connection-manager] - id=0x7f03e90db220: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
[INFO] [2022-08-11T14:56:01Z] [00007f035f7fe700] [connection-manager] - id=0x7f03e90db220: Requesting 1 new connections from http
[DEBUG] [2022-08-11T14:56:01Z] [00007f035f7fe700] [dns] - id=0x7f03e8ee2000: Host resolution requested for 10.255.0.18
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static: resolving host 10.255.0.18
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static: resolved record: 10.255.0.18
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static, resolving host 10.255.0.18 successful, returned 1 addresses
[INFO] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static: promoting spotty record 10.255.0.18 for 10.255.0.18 back to good list
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static: purging address 10.255.0.18 for host 10.255.0.18 from the cache due to cache eviction or shutdown
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static, invoking resolution callback for host 10.255.0.18 with 1 addresses
[DEBUG] [2022-08-11T14:56:02Z] [00007f02da7fc700] [dns] - static, invoking resolution callback for host 10.255.0.18 with 1 addresses
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [task-scheduler] - id=0x7f02b40015a0: Scheduling attempt_connection task for immediate execution
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [task-scheduler] - id=0x7f02b40015a0: Running attempt_connection task with <Running> status
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [socket] - id=0x7f033c000ec0 fd=184: initializing with domain 0 and type 0
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [socket] - id=0x7f033c000ec0 fd=184: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [socket] - id=0x7f033c000ec0 fd=184: beginning connect.
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [socket] - id=0x7f033c000ec0 fd=184: connecting to endpoint 10.255.0.18:443.
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [task-scheduler] - id=0x7f033c001510: Scheduling (null) task for future execution at time 30716563787478
[ERROR] [2022-08-11T14:56:02Z] [00007f035cff9700] [socket] - id=0x7f033c000ec0 fd=184: connection failure
[DEBUG] [2022-08-11T14:56:02Z] [00007f035cff9700] [channel-bootstrap] - id=0x7f03e8eed5e0: client connection on socket 0x7f033c000ec0 completed with error 1047.
Expected Behavior
Successful connection and successful transfer.
Current Behavior
Connection rejected and failed transfer.
Reproduction Steps
Deploy Minio in its simpler form and upload some data to some bucket. Try to use latest transfer manager with it to upload or download.
Possible Solution
At some point, the code seems to try to use https instead of the protocol and port got from the used endpoint.
Additional Information/Context
No response
AWS Java SDK version used
2.17.250
JDK version used
17.0.4
Operating System and version
Ubuntu 20.04.4
Hello @yetao ,
Thank you very much for reaching out. Could you please provide the code snippet associated with this issue?
Best,
Yasmine
This happens for me, too. The reproduction is pretty simple:
To setup minio:
- Start minio:
docker run --rm -it -p9000:9000 -p9002:9002 minio/minio server /data --console-address :9002 - Login to the console on
http://localhost:9002, usernameminioadmin, passwordminioadmin - Create a bucket named
test-bucket - Run the repro code below (I'm using version
2.17.253-PREVIEW)
public static void main(String[] args) throws ExecutionException, InterruptedException {
var transferManager = S3TransferManager.builder()
.s3ClientConfiguration(c -> c.endpointOverride(URI.create("http://localhost:9000"))
.credentialsProvider(() -> AwsBasicCredentials.create("minioadmin", "minioadmin"))
.region(Region.US_EAST_1))
.build();
var upload = transferManager.upload(ub -> ub
.putObjectRequest(b -> b.bucket("test-bucket")
.contentLength(1L)
.key("testfile.txt"))
.requestBody(AsyncRequestBody.fromBytes(new byte[] { 1 })));
upload.completionFuture().get();
}
The output is as follows:
Exception in thread "main" java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at Scratch.main(scratch_4.java:23)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:97)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:77)
at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:20)
Process finished with exit code 1
Exactly, just adding a download example and using access/secret kind of credentials:
public static void main(String args[]) throws Exception {
Log.initLoggingToStdout(LogLevel.Debug);
AwsBasicCredentials credentials = AwsBasicCredentials.create("KOQ72jCb2YpnjrOa", "qYq0EzPqgU9x1jqka5STRDnz1COLvcdq");
StaticCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(credentials);
S3ClientConfiguration.Builder s3ConfigBuilder = S3ClientConfiguration.builder()
.region(Region.of("eu-west-1"))
.endpointOverride(new URI("http://10.255.0.14:9000"))
.credentialsProvider(credentialsProvider);
S3TransferManager transferManager = S3TransferManager.builder()
.s3ClientConfiguration(s3ConfigBuilder.build()).build();
S3ClientBuilder s3Builder = S3Client.builder()
.credentialsProvider(credentialsProvider)
.region(Region.of("eu-west-1"))
.endpointOverride(new URI("http://10.255.0.14:9000"));
S3Client s3Client = s3Builder.build();
ListObjectsV2Request listObjectsBuilder = ListObjectsV2Request
.builder()
.bucket(bucket)
.build();
ListObjectsV2Iterable res = s3Client.listObjectsV2Paginator(listObjectsBuilder);
res.contents().stream().forEach(object -> downloadObject(transferManager, object));
}
private static void downloadObject(S3TransferManager transferManager, S3Object object) {
System.out.println("Will try to download: " + object.key());
Download<ResponseBytes<GetObjectResponse>> download =
transferManager.download(d -> d.getObjectRequest(g -> g.bucket(bucket).key(object.key()))
.responseTransformer(AsyncResponseTransformer.toBytes()));
download.completionFuture().join();
}
In addition, I have tried to test it using HTTPS, so I have deployed minio as follows:
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-08-08T18-34-09Z (go1.18.5 linux/amd64)
Status: 1 Online, 0 Offline.
API: https://192.168.1.131 https://10.255.0.14 https://10.192.4.4 https://127.0.0.1
RootUser: minioadmin
RootPass: minioadmin
Using the same code, but adapting the host to 'https//:127.0.0.1' results in the same exception, but different log and different reasons:
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [task-scheduler] - id=0x7f75f01020e0: Running socket_handler_close task with <Running> status
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [channel] - id=0x7f75f0001190: handler 0x7f75f0102040 shutdown in write dir completed.
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [task-scheduler] - id=0x7f75f00011b8: Scheduling (null) task for immediate execution
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [task-scheduler] - id=0x7f75f00011b8: Running (null) task with <Running> status
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [channel-bootstrap] - id=0x7f76d0cebb00: channel 0x7f75f0001190 shutdown with error 1029.
[ERROR] [2022-08-17T09:34:36Z] [00007f76117fa700] [http-connection] - static: Client connection failed with error 1029 (AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE).
[WARN] [2022-08-17T09:34:36Z] [00007f76117fa700] [connection-manager] - id=0x7f76d10cfbe0: Failed to obtain new connection from http layer, error 1029(TLS (SSL) negotiation failed)
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [connection-manager] - id=0x7f76d10cfbe0: Failing excess connection acquisition with error code 1029
[DEBUG] [2022-08-17T09:34:36Z] [00007f76117fa700] [connection-manager] - id=0x7f76d10cfbe0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] [2022-08-17T09:34:36Z] [00007f76117fa700] [connection-manager] - id=0x7f76d10cfbe0: Failed to complete connection acquisition with error_code 1029(TLS (SSL) negotiation failed)
[ERROR] [2022-08-17T09:34:36Z] [00007f76117fa700] [S3Endpoint] - id=0x7f76d10c33b0: Could not acquire connection due to error code 1029 (TLS (SSL) negotiation failed)
Please note that S3Client is still able to connect to the same endpoint. It connects to it with http or https without any problem. I can successfully see the following message from the previous sample code in all cases:
[INFO] [2022-08-17T09:34:22Z] [00007f75ccff97Will try to download: AutoSimple 2022-08-11 01.42.24/Himenaeos.doc
Thanks.
Best regards, Jorge
Seems this might be caused by #3315 , as it works for me on 2.17.236, but not 2.17.237 or any later version.
Hello @spmason , @yetao , @bpiper ,
Thank you very much for sharing the reproducible code sample. I have been able to reproduce the behavior you described. As mentioned by @bpiper , it seem that this behavior is only present on v2.17.237 and later and not present on v2.17.236. While we are looking into this issue are you able to use v2.17.236 or earlier version?
Thank you very much for reporting this!
Best,
Yasmine.
Hello @yasminetalby,
Thanks for your advice. Indeed 2.17.236 version improves the situation, but I'm still finding some inconveniences:
- It works for download and upload if I use http for the generic endpoint
- Upload works if I use https
- Download does not work if I use https
To run minio with https we need to put the certificate and the key in the right folder, then import it in the SDK (if it's a self-signed cert) and run the service, for example like this:
sudo ./minio server /home/jorge/minio-data/ --address :443
# ls -l /root/.minio/certs/
total 2612
drwx------ 2 root root 4096 ago 11 17:57 CAs
-rwxr-xr-x 1 root root 2654208 ago 11 18:09 certgen-linux-amd64
-rw-r--r-- 1 root root 5299 ago 11 18:09 main.zip
-rw------- 1 root root 241 ago 11 18:09 private.key
-rw-r--r-- 1 root root 765 ago 11 18:09 public.crt
I'm putting below what I can see from the debug log:
[INFO] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [AuthSigning] - (id=0x7f388c0013c0) Signing successfully built canonical request for algorithm SigV4, with contents
GET
/testbacula/AutoSimple%202022-08-22%2010.50.42/Atomorum.ppt
amz-sdk-invocation-id:9d9a8036-cd2a-4c7b-faec-27dff10f9efa
amz-sdk-request:attempt=1; max=1
content-length:0
host:127.0.0.1
range:bytes=0-8388607
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20220822T085651Z
amz-sdk-invocation-id;amz-sdk-request;content-length;host;range;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
[INFO] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [AuthSigning] - (id=0x7f388c0013c0) Signing successfully built string-to-sign via algorithm SigV4, with contents
AWS4-HMAC-SHA256
20220822T085651Z
20220822/eu-west-1/s3/aws4_request
09660e8b3e14ebb9b9aab95efe20c0d7bf176668f9dc4db101261a2b32072186
[INFO] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [AuthSigning] - (id=0x7f388c0013c0) Http request successfully built final authorization value via algorithm SigV4, with contents
AWS4-HMAC-SHA256 Credential=KOQ72jCb2YpnjrOa/20220822/eu-west-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;host;range;x-amz-content-sha256;x-amz-date, Signature=bbe0d909a12e8e86388de6eef03671ff145f08f1c2e06bedbfd724344483f8b9
[DEBUG] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [connection-manager] - id=0x7f39791672a0: Acquire connection
[DEBUG] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [connection-manager] - id=0x7f39791672a0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
[INFO] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [connection-manager] - id=0x7f39791672a0: Requesting 1 new connections from http
[DEBUG] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [dns] - id=0x7f3978cf8260: Host resolution requested for 127.0.0.1
[DEBUG] [2022-08-22T08:56:51Z] [00007f38ca7fc700] [dns] - id=0x7f3978cf8260: cached entries found for 127.0.0.1 returning to caller.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f388c006760: Scheduling attempt_connection task for immediate execution
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f388c006760: Running attempt_connection task with <Running> status
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: initializing with domain 0 and type 0
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: setting socket options to: keep-alive 0, keep idle 0, keep-alive interval 0, keep-alive probe count 0.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: beginning connect.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: connecting to endpoint 127.0.0.1:443.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4000f90: Scheduling (null) task for future execution at time 9443584253598
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001010: Scheduling epoll_event_loop_unsubscribe_cleanup task for immediate execution
[INFO] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: connection success
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket] - id=0x7f38c40010b0 fd=180: assigning to event loop 0x7f3978cfdac0
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel-bootstrap] - id=0x7f3978d03190: client connection on socket 0x7f38c40010b0 completed with error 0.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: Beginning creation and setup of new channel.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001360: Scheduling on_channel_setup_complete task for immediate execution
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001010: Running epoll_event_loop_unsubscribe_cleanup task with <Running> status
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001360: Running on_channel_setup_complete task with <Running> status
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: setup complete, notifying caller.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: no message pool is currently stored in the event-loop local storage, adding 0x7f38c4001760 with max message size 262144, message count 4, with 4 small blocks of 128 bytes.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel-bootstrap] - id=0x7f3978d03190: channel 0x7f38c4001460 setup succeeded: bootstrapping.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [socket-handler] - id=0x7f38c41023c0: Socket handler created with max_read_size of 262144
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4102520: Scheduling tls_timeout task for future execution at time 9450584481583
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001250: Scheduling socket_written_task task for immediate execution
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001250: Running socket_written_task task with <Running> status
[WARN] [2022-08-22T08:56:51Z] [00007f38e27fc700] [tls-handler] - id=0x7f38c41024f0: negotiation failed with error Certificate is untrusted (Error encountered in /codebuild/output/src856241491/src/aws-crt-java/crt/s2n/tls/s2n_server_cert.c:49)
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel-bootstrap] - id=0x7f3978d03190: tls negotiation result 1029 on channel 0x7f38c4001460
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001660: Scheduling channel_shutdown task for immediate execution
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: Channel shutdown is already pending, not scheduling another.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001660: Running channel_shutdown task with <Running> status
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: beginning shutdown process
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: handler 0x7f38c41023c0 shutdown in read dir completed.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [tls-handler] - id=0x7f38c41024f0: Shutting down read direction with error code 1029
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [channel] - id=0x7f38c4001460: handler 0x7f38c41024f0 shutdown in read dir completed.
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001488: Scheduling (null) task for immediate execution
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c4001488: Running (null) task with <Running> status
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [tls-handler] - id=0x7f38c41024f0: Scheduling delayed write direction shutdown
[DEBUG] [2022-08-22T08:56:51Z] [00007f38e27fc700] [task-scheduler] - id=0x7f38c41026c0: Scheduling s2n_delayed_shutdown task for future execution at time 9466397546484
It says 'untrusted certificate'. However, as I said, upload does not present this problem, neither using S3Client, so apparently it's just transfer manager download code what presents some problem with generic enpoints, https and version 2.17.236.
To reproduce the issue, the same code I provided earlier can be used, just changing the endpoint from http://10.255.0.14:9000 to https://10.255.0.14 (just to avoid confusion, in the debug log I have provided I'm using https://127.0.0.1, but that is pointing to the same host and service).
Best, Jorge
Hello @yetao , @spmason , @bpiper
Thank you very much for the detailed documentation and information regarding this issue. The behavior described in this submission seems to be originated from the CRT client which is used by the S3TransferManager in AWS Java SDK v2. We brought this up to the CRT team and are working in collaboration with them to solve this issue.
I will post an update as soon as the behavior is fixed.
Best,
Yasmine.
Thanks @yasminetalby
It looks like this started working for me in 2.17.264 and continues to work on the latest version. Following the links on this issue through from #3370 it looks like #3403 might have fixed it? I can't see much that appears relevant in the Changelog, though
Hello @spmason ,
I am happy to hear that you can now get the expected behavior!
#3403 would not appear in the change log as it has been merged after the release of v2.17.264 and is not yet released this is why you are not seeing any related informations in the changelogs.
#3384 passes endpoint override to S3 CRT Async HTTP client.
Thank you very much for bringing this issue up to our attention. Let me know if you have any other questions or if this issue can be closed.
Best,
Yasmine
Hello @yasminetalby,
On my side, using latest version 2.17.272 I get the same behavior that I explained in my last comment here: https://github.com/aws/aws-sdk-java-v2/issues/3350#issuecomment-1222063324
Basically:
- It works if the endpoint is using HTTP
- It does not work if the endpoint is using HTTPS (downloads, uploads do work with HTTPS).
Not sure if you would prefer to track that in a different issue.
Thanks. Best regards, Jorge
Hello @yetao ,
Thank you very much for bringing this up. It seems that the issue is only experienced with S3TransferManager because it is also related to the usage of the CRTClient. We are working on investigating the origin of the behavior with CRT. We can keep this issue open for the behavior described in #3350comment based on your preferences.
Best,
Yasmine.
Hello @yetao ,
Could you please provide more detail on your approach using https endpoints as well as provide the code sample for both upload and download using https that leads to the successful behavior for upload and failure behavior for download.
Thank you very much for your collaboration. Sincerely,
Yasmine
Hi Yasmin
I am facing this below issue and tried with multiple version of S3 Transfer manager (2.17.236, 264, 272, 273,-preview) versions. When Running this code from the AWS Lambda getting the error as below
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
I am using the below code just for reference.
public static S3TransferManager s3TransferManager() {
long mb = 1024;
S3TransferManager tm = S3TransferManager.builder()
.s3ClientConfiguration(cfg -> cfg.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.region(Region.AP_SOUTHEAST_1).minimumPartSizeInBytes(10 * mb))
.build();
return tm;
}
FileUpload upload = transferManager
.uploadFile(u -> u.source(tempZipFile)
.putObjectRequest(p -> p.bucket(bucketName)
.contentType(ZIP_CONTENT_TYPE).metadata(metadata).key(objectKey + zippedFolderName)));
upload.completionFuture().join();
Also when I run this piece of code from main() method I am able to connect to S3 bucket and uploads the files successfully. The error occurs only when it is running via lambda
Hello @kenduraghav ,
Thank you very much for reaching out. The exception you mention is relative to number of retries. This means that you have encountered a retryable exception and upon retry have reach the maximum number of retry of the request. You can learn more about retry behavior with AWS SDKs here. Given solely this error message, it does not give much context on what is happening here. Could you please provide full error logs to investigate what exception caused this retry. Best,
Yasmine
Hi Yasmine,
Thanks for the info. I will go through the retry behaviour on the shared link.
Please find the full stack trace as below.
Note I have initially excluded the below dependencies from S3 which was commented out at later stage. Even after that I got the same error as below.
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the maximum number of retries has been exceeded.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:97)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:77)
at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:20)
Hello @kenduraghav,
Thank you very much for providing the full stack trace. This seems to also be an exception from the CRT S3 client. Could you enable CRT logs? You can do so by adding the following code before transfer manager initialization:
// logging to a file
Log.initLoggingToFile(Log.LogLevel.Debug, "log.txt")
// logging to the console
Log.initLoggingToStdout(Log.LogLevel.Debug);
Thank you very much for your time and collaboration.
Best,
Yasmine
Hello @ionutss ,
Thank you very much for your additional feedback. Are you having the exact same use case as @kenduraghav?
Best,
Yasmine
Hi @yasminetalby, I added the CRT logs as you mentioned above, and it looks like the issue is generated by a proxy on my machine:
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [standard-retry-strategy] - id=0x7fb3c6d1a670: token acquired callback invoked with error Success. with token 0x7fb2d40012c0 and nested token 0x7fb2d4001600
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: Acquire connection
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
[INFO] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: Requesting 1 new connections from http
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [http-connection] - http_proxy environment found
[ERROR] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [http-connection] - Could not parse found proxy URI.
[ERROR] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: http connection creation failed with error code 36(An input string was passed to a parser and the string was incorrectly formatted.)
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: Failing excess connection acquisition with error code 36
[WARN] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [connection-manager] - id=0x7fb3c63ff510: Failed to complete connection acquisition with error_code 36(An input string was passed to a parser and the string was incorrectly formatted.)
[ERROR] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [S3Endpoint] - id=0x7fb3c665a130: Could not acquire connection due to error code 36 (An input string was passed to a parser and the string was incorrectly formatted.)
[DEBUG] [2022-09-23T10:02:02Z] [00007fb2c7fff700] [S3Client] - id=0x7fb3c6401100 Client scheduling retry of request 0x7fb2d40013a0 for meta request 0x7fb3c6658240 with token 0x7fb2d40012c0.
Hi Yasmine
Sure I will enable the logging and let you know further
Regards Raghav
On Fri, 23 Sep 2022 at 06:08, Yasmine Talby @.***> wrote:
Hello @kenduraghav https://github.com/kenduraghav,
Thank you very much for providing the full stack trace. This seems to also be an exception from the CRT S3 client. Could you enable CRT logs? You can do so by adding the following code before transfer manager initialization:
// logging to a file
Log.initLoggingToFile(Log.LogLevel.Debug, "log.txt")
// logging to the console Log.initLoggingToStdout(Log.LogLevel.Debug);
Thank you very much for your time and collaboration.
Best,
Yasmine
— Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-java-v2/issues/3350#issuecomment-1255605739, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKZJPXUNIS77VLIHJHH46XDV7TKG7ANCNFSM56ISZRWA . You are receiving this because you were mentioned.Message ID: @.***>
-- Thanks & Regards, Raghav
Since I left my previous organisation, I will try my best to check that. Regards Raghavendran On Sat, 4 Feb 2023 at 06:13, Zoe Wang @.***> wrote:
We have released a fix. Could you try with the latest version?
— Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-java-v2/issues/3350#issuecomment-1416471503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKZJPXSBEVY4APVLS2UHFULWVV7J5ANCNFSM56ISZRWA . You are receiving this because you were mentioned.Message ID: @.***>
-- Thanks & Regards, Raghav
Closing, since a fix for the original issue was released.
Please open a fresh new issue if you are still experiencing this error.
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
