aws.getCallerIdentity() doesn't work with skipMetadataApiCheck or skipCredentialsValidation

[michaeldop]

What happened?

After upgrading from 5.13 to the latest (5.18) we can no longer run pulumi up in our CI environment. The AWS creds are provided through the EC2 instance role. I have tried both setting the stack state aws:skipMetadataApiCheck false and aws:skipCredentialsValidation true as well as the new ENV vars AWS_SKIP_METADATA_API_CHECK and AWS_SKIP_CREDENTIALS_VALIDATION to no avail.

Maybe we are missing some other configuration or setup.

Steps to reproduce

call aws.getCallerIdentity() from a program

Expected Behavior

pulumi up can run normally without an error

Actual Behavior

Error: invocation of aws:index/getCallerIdentity:getCallerIdentity returned an error: 1 error occurred: * error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

Please see https://registry.terraform.io/providers/hashicorp/aws for more information about providing credentials.

Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable

Output of pulumi about

@pulumi/aws 5.18.0 @pulumi/pulumi 3.43.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[danielrbradley]

Thank you @michaeldop for reporting this issue. Some preliminary searching found this issue in the upstream provider which seems it might be describing the same issue you're facing: https://github.com/hashicorp/terraform-provider-aws/issues/26074

It's likely this will need to be solved upstream before we can pull the fix.

[michaeldop]

🤔 I don't think this error is related to that specific issue, but I could be wrong. I am not seeing a metadata error or timeout occur when getting the credentials. I am also using the default aws provider with region and assumeRole configured.

Are getting credentials from EC2 instance profile not supported anymore with the default provider?

I thought maybe these PRs would help my issue but I still observe the same behavior https://github.com/pulumi/pulumi-aws/pull/2148 https://github.com/pulumi/pulumi-aws/pull/2149

[michaeldop]

I also wanted to add when I enable debugging I see the role being assumed properly but still get the same error. In order for the role to be assumed I set this ENV var AWS_SKIP_METADATA_API_CHECK=false

[rdanno]

I bet we are talking about the same issue here https://github.com/pulumi/pulumi-aws/issues/2194

Does this only occur on STS v2 enabled regions? Does it work with regular user keys?

[mikhailshilkov]

This issue has been quiet for two years and a suspicious duplicate https://github.com/pulumi/pulumi-aws/issues/2194 was closed. I'll go ahead and close it too - if anyone still experiences the problem, please open a new issue with a repro.