aws-sdk-java-v2 icon indicating copy to clipboard operation
aws-sdk-java-v2 copied to clipboard
verified publisher icon aws

Configuration sharing from SDK client to credentials provider

Open david-katz opened this issue 7 years ago • 3 comments

Created a LambdaClient as follows:

`

 httpClientBuilder =
    ApacheHttpClient.builder()
        .proxyConfiguration(
            proxyConfig
                .endpoint(URI.create("http://myproxy.com:8080"))
                .username("username")
                .password("password")
                .build());

client =
    LambdaClient.builder()
        .httpClientBuilder(httpClientBuilder)
        .region(region)
        .build();

`

In SdkDefaultClientBuilder being called internally from DefaultStsClientBuilder, I see that the httpClient is being created with an empty proxyConfiguration.

Expected Behavior

Sts should use the same proxyConfigration as the one I passed to the LambdaClient.

Current Behavior

Proxy is not used, and thus the host is not resolvable.

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: sts.amazonaws.com at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException(RetryableStage.java:164) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:117) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:67) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:47) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:41) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:25) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:272) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:90) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:102) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:67) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:239) at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:68) at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:68) at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:133) at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:90) at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:78) at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.resolveCredentials(StsAssumeRoleCredentialsProvider.java:42) at software.amazon.awssdk.services.sts.internal.StsProfileCredentialsProviderFactory$StsProfileCredentialsProvider.resolveCredentials(StsProfileCredentialsProviderFactory.java:93) at software.amazon.awssdk.awscore.internal.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:54) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:65) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:239) at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:68) at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:68) at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:133) at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:90) at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:78) at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.resolveCredentials(StsAssumeRoleCredentialsProvider.java:42) at software.amazon.awssdk.services.sts.internal.StsProfileCredentialsProviderFactory$StsProfileCredentialsProvider.resolveCredentials(StsProfileCredentialsProviderFactory.java:93) at software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider.resolveCredentials(ProfileCredentialsProvider.java:110) at software.amazon.awssdk.awscore.internal.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:54) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:65) at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) at software.amazon.awssdk.services.lambda.DefaultLambdaClient.invoke(DefaultLambdaClient.java:1078) at com.mytest.ITLambda.tes(ITLambda.java:58) Caused by: java.net.UnknownHostException: sts.amazonaws.com at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:80) at com.sun.proxy.$Proxy0.connect(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:200) at software.amazon.awssdk.http.apache.ApacheHttpClient.access$400(ApacheHttpClient.java:92) at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:178) at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:175) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:58) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:50) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:34) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:205) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:139) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:109) ... 57 more

Possible Solution

Pass proxy configuration to Sts client.

  • AWS Java SDK version used: 2.0.0-preview-12
  • JDK version used:1.8.0_171
  • Operating System and version: MacOS 10.13.6

david-katz avatar Oct 06 '18 08:10 david-katz

Hi @david-katz, If you define your proxy settings as system properties, (and if you're using the Apache HTTP client, which is the default for sync clients), any client that gets constructed after they're set should pick up the settings, including the STS client used by the StsAssumeRoleCredentialsProvider.

System.setProperty("http.proxyHost", PROXY_HOST);
System.setProperty("Http.proxyPort", PROXY_PORT);

LambdaClient lambda = LambdaClient.create(); // should make all requests through the proxy.

dagnir avatar Oct 12 '18 16:10 dagnir

What if I need authentication for the proxy is this supported? When I try the following exception shows:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request:  http://usr:[email protected]
Caused by: java.net.UnknownHostException: http://usr:[email protected]

Strange that you have a test testProxyConfigurationWithSystemPropertyEnabled but im not able to duplicate it. Given that I do:

        System.setProperty("http.proxyPort", "port");
        System.setProperty("http.proxyHost", "http://host.com");
        System.setProperty("http.proxyUser", "usr");
        System.setProperty("http.proxyPassword", "pwd");
        ProxyConfiguration config = ProxyConfiguration.builder().build();

        S3Client.create();

The config object only has host and post set, user and pwd are null.

favetelinguis avatar Jan 18 '19 06:01 favetelinguis

Can this be marked as a bug instead of a feature? Asking a client to unconditionally use a proxy (using a public API designed specifically for that purpose) and the client not respecting that setting is a bug. Wanting the publicly documented behavior to operate correctly is not a feature request.

I'd actually argue that this is a pretty serious bug since it means that proxy configuration is completely ignored for some calls. What makes it particularly interesting is if you use STS directly, it apparently ignores the configuration even if it doesn't need to do anything behind-the-scenes to fetch credentials (code sample below).

Note that there are two workarounds:

  • Use System Properties - this doesn't work for us (we need those properties for other things because we have different properties for AWS, plus those can't be managed through the same configuration system that manages everything else in our applications)
  • Use the routePlanner which for some reason is respected (this works, but is complex)
    ProxyConfiguration proxyConfiguration = ProxyConfiguration
        .builder()
        .endpoint(URI.create(PROXY_SERVER))
        .nonProxyHosts(Collections.emptySet())
        .useSystemPropertyValues(false)
        .build();

    ApacheHttpClient.Builder httpClientBuilder = ApacheHttpClient
        .builder()
        .proxyConfiguration(proxyConfiguration);

    StsClient client = StsClient.builder()
        .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY_ID, SECRET_ACCESS_KEY)))
        .region(REGION)
        .httpClientBuilder(httpClientBuilder)
        .build();

    GetCallerIdentityResponse response = client.getCallerIdentity();

abatkin avatar May 26 '20 19:05 abatkin