Unable to connect to local Minio server when a proxy configuration is specified, even when the Minio server host is set in the http.nonProxyHosts.

[armlesshobo]

Describe the bug

With an async client built with the following code, ...

S3AsyncClient.crtBuilder()
                .httpConfiguration(
                    S3CrtHttpConfiguration.builder()
                        .proxyConfiguration(
                            S3CrtProxyConfiguration.builder()
                                .build()
                        )
                        .build()
                )
                .credentialsProvider(credProvider)
                .region(region)
                .endpointOverride(endpointUri) // URI.create("http://10.193.72.200:9000")
                .build(); 

and with the following JVM properties set:

 -Dhttp.proxyHost=10.193.0.95  -Dhttp.proxyPort=80 -Dhttp.nonProxyHosts=10.193.72.200

I cannot connect to an S3 service running on the local network, despite setting the http.nonProxyHosts property

Expected Behavior

I expect to be able to connect to both an S3 service running on the local network, as well as to AWS S3, when HTTP proxy settings are provided as JVM options.

Current Behavior

It appears the HTTP request is being sent through the proxy, instead of being sent through the local network.

I see this in the log:

software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Proxy-based connection establishment failed because the CONNECT call failed

Reproduction Steps

Run a local instance of minio and set up a region/bucket/access key in the WebUI.

Build a client with the following code:

var asyncClient = S3AsyncClient.crtBuilder()
                .httpConfiguration(
                    S3CrtHttpConfiguration.builder()
                        .proxyConfiguration(
                            S3CrtProxyConfiguration.builder()
                                .build()
                        )
                        .build()
                )
                .credentialsProvider(credProvider)
                .region(region)
                .endpointOverride(endpointUri) // URI.create("<your_minio_server_host>:<port>")
                .build(); 

run the program with these JVM options:

 -Dhttp.proxyHost=10.193.0.95  -Dhttp.proxyPort=80 -Dhttp.nonProxyHosts=<your_minio_server_host>

(It actually doesn't matter if the http.proxyHost is set to a real HTTP proxy host, because we shouldn't be going through it anyway.)

Perform any action with that client. I tried to list buckets in my test.

        try {
            asyncClient.listBuckets().get()
                    .buckets()
                    .forEach(bucket -> logger.info("--> {}", bucket.name()));
        } catch (Exception ex) {
            logger.error("Unable to list available buckets: {}", ex.getMessage());
        }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.23.3

JDK version used

openjdk version "17.0.11" 2024-04-16 LTS

Operating System and version

Windows 11 Enterprise Version 10.0.22621

[steveloughran]

those http system property settings are picked up by the java.net httpclient; aws sdk uses apache httpclient which has never picked them up.

[debora-ito]

@armlesshobo I'm sorry for long delay in replying here.

I tested a more recent SDK version 2.27.5/aws-crt 0.30.6, and the proxy system properties are being honored by the S3 CRT based http client - the crt logs of my local tests show the request host host:10.193.72.200:9000:

[INFO] [2024-08-19T21:48:44Z] [0000000174b13000] [AuthSigning] - (id=0x6000030880a0) Signing successfully built canonical request for algorithm SigV4, with contents 
GET
/

amz-sdk-invocation-id:xxx
amz-sdk-request:attempt=1; max=1
content-length:0
host:10.193.72.200:9000
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240819T214844Z
x-amz-security-token:xxx

amz-sdk-invocation-id;amz-sdk-request;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
UNSIGNED-PAYLOAD

Can you try a more recent version of the SDK? If the issue still persists after the upgrade, please share the CRT Trace logs, instructions can be found in our Dev Guide - https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html

@steveloughran all the http clients supported by the SDK will honor proxy system properties - that's the expectation at least. If the latest SDK version is not picking them up that's a bug.

[steveloughran]

all the http clients supported by the SDK will honor proxy system properties - that's the expectation at least. If the latest SDK version is not picking them up that's a bug.

@debora-ito really? good to know. and means some more things to get from failure logs

[pingw33n]

The problem appears to be here where the proxy is resolved once per HTTP client instance while it should be done per request. Also CrtConfigurationUtils.resolveProxy() mistakenly uses proxy host to test against nonProxyHosts while it should test the request's host.

[ben-ganley]

We are also running into this issue.

We have tried to test out the CRT client however found it doesn't respect the nonProxyHosts. We are using the latest version of the CRT client at the time of writing.

[debora-ito]

For anyone experiencing this issue: please send us the CRT trace logs and we will take a look. Make sure to remove any sensitive data from the logs, like access keys.