aws
Auth: DefaultCredentialsProvider uses InstanceProfileCredentials instead of WebIdentityTokenFileCredentials
Describe the bug
DefaultCredentialsProviderChain returns credentials from InstanceProfileCredentialsProvider even though AWS_WEB_IDENTITY_TOKEN_FILE env variable is set and WebIdentityTokenFileCredentialsProvider is expected to be used to resolveCredentials
As WebIdentityTokenFileCredentialsProvider is logged at DEBUG level, this failure is hidden from users and user ends up spending valuable time debugging why the credentials are not working as expected
23/04/08 05:11:03 **DEBUG** AwsCredentialsProviderChain: Unable to load credentials from WebIdentityTokenCredentialsProvider(): Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath
Expected Behavior
when web identity token file is provided, provider chain should fail with exception and actionable exception message when credentials cannot be read from the WebIdentityTokenFileCredentialsProvider. Provider chain should stop processing further
Current Behavior
Default provider chain ignores the fatal WebIdentityTokenCredentialsProvider due to multiple http implementations and returns instance profile credentials which is not expected. As WebIdentityTokenCredentialsProvider's failure are logged as DEBUG logs, its hard for users to figure out why the expected web identity token credentials were not used and where the unexpected credentials are being returned from.
23/04/08 05:11:03 DEBUG AwsCredentialsProviderChain: Unable to load credentials from SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
23/04/08 05:11:03 DEBUG AwsCredentialsProviderChain: Unable to load credentials from EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
**23/04/08 05:11:03 DEBUG AwsCredentialsProviderChain: Unable to load credentials from WebIdentityTokenCredentialsProvider(): Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading implementations, please explicitly provide an HTTP client via the client builders, set the software.amazon.awssdk.http.service.impl system property with the FQCN of the HTTP service to use as the default, or remove all but one HTTP implementation from the classpath**
23/04/08 05:11:03 DEBUG AwsCredentialsProviderChain: Unable to load credentials from ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.
23/04/08 05:11:03 DEBUG AwsCredentialsProviderChain: Loading credentials from InstanceProfileCredentialsProvider()
Reproduction Steps
- create a jar which has multiple http implementations
- initialize s3 client with default credentials provider
- access a s3 file which only the role in webIdentityTokenFile has access to
- set role arn for token file
- run the jar on ec2 instance which has
webIdentityTokenFile
Possible Solution
- WebIdentityTokenFileCredentialsProvider should fail fast with fatal exception when create fails due to multiple http implementations
- log the failure in WebIdentityTokenFileCredentialsProvider at ERROR level so users see the failure and can take action
Additional Information/Context
No response
AWS Java SDK version used
2.17.257
JDK version used
1.8
Operating System and version
linux
Hi @abmo-x this is not a bug, this is how the DefaultCredentialsProviderChain was designed. Any failure in one individual credential provider in the chain will be logged and the next provider will be called.
We won't be able to approve your PR because (1) we don't intend to make the whole credential chain fail if WebIdentityTokenFileCredentialsProvider fails, and (2) changing this behavior is a breaking change.
If you want the WebIdentityTokenFileCredentialsProvider to fail fast we recommend you set WebIdentityTokenFileCredentialsProvider explicitly as the credential provider, as showed here in the Java SDK Developer Guide - https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-specify.html
Hi @debora-ito Thanks for your feedback. I understand this is a breaking change. Unfortunately your proposal to set the provider chain explicitly is not always feasible, especially in scenarios where we rely on some SDK which initializes the aws client internally.
would you be open to logging the http implementations error as a warn log instead of debug and not throw the exception? That will help users at least be aware why the webIdentity token was not used and avoid lot of engineering debug cycles. Let me know your thoughts.
The webIdentity provider failing silently as it does current is not really helpful. If you have a different suggestion or recommendation, let me know and I can update the PR accordingly.
Another thought, if multiple http implementations on the class path is a fatal SDK failure, do we want to check for this earlier when the sdk initializes? maybe we can move this verification out of the provider chain to not break the provider chain design.
@abmo-x sorry for the long silence.
Discussed this issue with the team, and instead of changing the log level for everything, we want to improve the logging in each credential provider, so that every time a credential fetch fails you have more meaningful and actionable error message.
Another thought, if multiple http implementations on the class path is a fatal SDK failure, do we want to check for this earlier when the sdk initializes? maybe we can move this verification out of the provider chain to not break the provider chain design.
Not all credential providers require the use of an HTTP client to send a request, so it only makes sense to make the checks in those credential providers that will use them.
can there be an environment variable to opt-in to failure - otherwise it silently falls back and then use the wrong (node) credentials forever instead of at least retrying in case of transient error
@debora-ito Any updates on this issue? We spent a bunch of time investigating why the expected credentials were not getting picked up again.