Slow requests to get credentials from InstanceProfileCredentialsProvider

[ZeevG]

I've noticed that the first request using SqsAsyncClient takes around 5000ms. This seems to be caused by InstanceProfileCredentialsProvider. The slow request seems to re-occur roughly every hour as the credential cache expires.

Stack trace of where 99% of time was spent in a slow request.

…net.www.protocol.http.HttpURLConnection.getInputStream (Unknown Source)
             java.net.HttpURLConnection.getResponseCode (Unknown Source)
…on.awssdk.regions.util.HttpResourcesUtils.readResource (HttpResourcesUtils.java:114)
…redentials.InstanceProfileCredentialsProvider.getToken (InstanceProfileCredentialsProvider.java:91)
…fileCredentialsProvider.getCredentialsEndpointProvider (InstanceProfileCredentialsProvider.java:76)
…credentials.HttpCredentialsProvider.refreshCredentials (HttpCredentialsProvider.java:74)
….amazon.awssdk.utils.cache.CachedSupplier.refreshCache (CachedSupplier.java:132)
  software.amazon.awssdk.utils.cache.CachedSupplier.get (CachedSupplier.java:89)
                                 java.util.Optional.map (Unknown Source)
…credentials.HttpCredentialsProvider.resolveCredentials (HttpCredentialsProvider.java:146)
…entials.AwsCredentialsProviderChain.resolveCredentials (AwsCredentialsProviderChain.java:91)
…internal.LazyAwsCredentialsProvider.resolveCredentials (LazyAwsCredentialsProvider.java:52)
…dentials.DefaultCredentialsProvider.resolveCredentials (DefaultCredentialsProvider.java:100)
…t.handler.AwsClientHandlerUtils.createExecutionContext (AwsClientHandlerUtils.java:71)
…t.handler.AwsAsyncClientHandler.createExecutionContext (AwsAsyncClientHandler.java:64)
software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.createExecutionContext…sdk.core.client.handler.BaseAsyncClientHandler.execute (BaseAsyncClientHandler.java:63)
…k.awscore.client.handler.AwsAsyncClientHandler.execute (AwsAsyncClientHandler.java:51)
….awssdk.services.sqs.DefaultSqsAsyncClient.sendMessage (DefaultSqsAsyncClient.java:1271)

I noticed that there is a similar issue for v1 of the java sdk. I am running this code from within a docker container, could my issue be related? https://github.com/aws/aws-sdk-java/issues/2171

Any thoughts would be awesome, thanks!

[debora-ito]

Hi @ZeevG what version of the SDK are you using?

[ZeevG]

Hi @debora-ito we're using version 2.10.41. It does look like there have been some changes to InstanceProfileCredentialsProvider recently. I'll try upgrading to 2.10.73.

[debora-ito]

Actually we are experiencing this too, it seems to be an issue on the service side. We'll reach out to the service team and will update when we hear back.

[ZeevG]

It looks like the issue is still present after upgrading to 2.10.73, although it is definitely less severe. Requests to update credentials take around 1 second instead of 5 seconds.

I'm going to upgrade versions and try enabling the asyncCredentialUpdateEnabled flag to mitigate the issue until this is resolved.

[zoewangg]

That's because we have reduced connectionTimeout and readtimeout to 1 sec in #1568 :)

[kwahsog]

We're getting exactly the same issue with the SES client. What's the recommended fix?

Will try turning on asyncCredentialUpdateEnabled, but presumably it will still error and generate ERROR logs when it times out?

Using SDK version 2.15.45.

Thanks!

[ffeltrinelli]

Hi! I'm also interested in this one. I'm having intermittent latency issues, reaching the non-configurable 1 sec read timeout multiple times per day, while the SDK is fetching credentials from IMDS. Not being able to configure a timeout shorter than 1 sec is problematic: my application is latency sensitive.

[boris-ait]

We are also facing the same latency issue. Any estimation of when it will be resolved?

[zichuanwang]

We are facing the same latency issue at Cardless

[armandobelardo]

@zoewangg or @debora-ito we are also facing the same issue, does enabling async token refresh (refreshCredentialsAsync ) solve this problem so it's not blocking the worker thread? We'll be testing shortly but seems like it could be a reasonable workaround?

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/InstanceProfileCredentialsProvider.html#InstanceProfileCredentialsProvider-boolean-

[ffeltrinelli]

Hi! I talked about this issue and described our custom solution in this article.

[debora-ito]

This issue was created a long time ago, and we haven't had any recent reports of high latency on the IMDS side, so I'll mark this to close soon.

If anyone is still experiencing this, please raise a fresh new github issue and provide the client side metrics you generated showing the credentials fetch duration.

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.