Configured Non Proxy Hosts in Java S3 Client does not appear to be honored

[fintam]

Describe the bug

I am running a java container that interfaces with S3 through the AWS API. There are times when the application (that lives in us-east-2) may need to talk to an s3 bucket that is not in it's deployed region. In this situation, we need to go through proxy.

So right now, we have some following code set in the application when configuring the S3 client. The line "setNonProxyHosts" code is supposed to prevent us from going through the proxy if url generated ends with that following string.

ClientConfiguration config = new ClientConfiguration();
//AWS_REGION = the applications deployed region or right now us-east-2
config.setNonProxyHosts(String.format("*.s3.%s.amazonaws.com", System.getenv("AWS_REGION")));
config.setProxyHost(System.getenv("PROXY_HOST"));
config.setProxyPort(Integer.parseInt(System.getenv("PROXY_PORT")));
config.setProxyUsername(System.getenv("service account username"));
config.setProxyPassword(System.getenv("service account password"));
EndpointConfiguration ec = new EndpointConfiguration("https://s3.amazonaws.com", region);
client = AmazonS3ClientBuilder.standard().withClientConfiguration(config).withEndpointConfiguration(ec).build();

However, what we are seeing is that everything is going through the proxy and nothing is being routed around it when this is set, which is making some of our stuff fail. Error given below.

com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: proxyurl.com), S3 Extended Request ID: null at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)

To be clear, I have tried a few different values to insert into "setNonProxyHosts". This is the list I have tried

• *.s3.us-east-2.amazonaws.com
• .s3.us-east-2.amazonaws.com
• s3.us-east-2.amazonaws.com

Expected Behavior

Expected behavior is that, when our application that is deployed to us-east-2 tries to interface with a s3 bucket located in us-east-2, that by setting the nonproxyhosts list, we will be able to avoid going through the proxy. This is very similar behavior to setting the ProxyBypassList in the .NET AWS S3 package.

Current Behavior

When we set the setNonProxyHosts property on ClientConfiguration, it doesn't appear what we are setting is being honored which is making everything go through the proxy.

Reproduction Steps

` private AmazonS3 client;

//assume AWS_REGION = us-east-2 private void configureClient(String region) { ClientConfiguration config = new ClientConfiguration(); //AWS_REGION = the applications deployed region or right now us-east-2 config.setNonProxyHosts(String.format("*.s3.%s.amazonaws.com", System.getenv("AWS_REGION"))); config.setProxyHost(System.getenv("PROXY_HOST")); config.setProxyPort(Integer.parseInt(System.getenv("PROXY_PORT"))); config.setProxyUsername(System.getenv("service account username")); config.setProxyPassword(System.getenv("service account password")); EndpointConfiguration ec = new EndpointConfiguration("https://s3.amazonaws.com", region); client = AmazonS3ClientBuilder.standard().withClientConfiguration(config).withEndpointConfiguration(ec).build(); }

configureClient("us-east-2"); return client.doesObjectExist("bucket name", "key of object in s3 bucket"); `

Possible Solution

This very well may be a misunderstanding of how this property works. Am I misunderstanding which wildcard character is supported or not supported? I have looked in the documentation and I don't seen anything about wildcard characters being supported here, but setting the list of nonproxyhosts this way we do in a few different places in our company. So I would expect this is supported here as well in some degree.

Additional Information/Context

No response

AWS Java SDK version used

1.12.126

JDK version used

docker.artifactory.sentry.com/amazoncorretto:11.0.15-alpine

Operating System and version

docker.artifactory.sentry.com/amazoncorretto:11.0.15-alpine

[debora-ito]

@fintam I'm sorry for the delayed response. Are you still seeing the issue?

Can you enable the verbose wirelogs to check if the s3 endpoint used in the request is really matching the nonProxyHosts pattern? For instructions on how to enable the wirelogs see: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#sdk-net-logging-verbose

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[SathyaDhana]

hi @fintam I am facing this issue now. Were you able to solve this problem in AWS SDK 1.x.

Everything works fine with AWS SDK 2.x.

Please advise.

[kekwan]

@debora-ito I am also hitting same issue with AWS SDK 1.12.172. Is this a known bug ? It's not easy for us to try to upgrade SDK versions.

If I wanted to bypass the proxy for a particular S3 bucket. e.g. s3://test-bucket-us-east-2. How should value for config.setNonProxyHosts be formatted?

[kekwan]

I was able to get it to work after turning on wire debugging. We had to add both <bucket-name>.s3.<region>.amazonaws.com and <bucket-name>.s3.amazonaws.com endpoints as non proxy hosts for it to work. Also, I don't think wildcards are enabled.