Unable to validate AWS credentials - using EC2 instance profile

[barryrobison]

What happened?

[2022-08-10T06:38:50Z]         [urn=urn:pulumi:dev-apse2-main::aws-eks::pulumi:providers:aws::dev-ap-southeast-2]
  | [2022-08-10T06:38:50Z]         profile                  : "dev"
  | [2022-08-10T06:38:50Z]         region                   : "ap-southeast-2"
  | [2022-08-10T06:38:50Z]       ~ skipCredentialsValidation: "false" => "true"
  | [2022-08-10T06:38:50Z]         skipGetEc2Platforms      : "true"
  | [2022-08-10T06:38:50Z]         skipMetadataApiCheck     : "false"
  | [2022-08-10T06:38:50Z]         skipRegionValidation     : "true"
  | [2022-08-10T06:38:50Z]       + skipRequestingAccountId  : "true"
  | [2022-08-10T06:38:50Z]       + stsRegion                : "ap-southeast-2"
  | [2022-08-10T06:38:50Z]       ~ version                  : "5.6.0" => "5.10.0"
  | [2022-08-10T06:38:52Z    ~ aws:iam/openIdConnectProvider:OpenIdConnectProvider: (update)
  | [2022-08-10T06:38:52Z]         [id=arn:aws:iam:::oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/zzzz]
  | [2022-08-10T06:38:52Z]         [urn=urn:pulumi:dev-apse2-main::aws-eks::aws:iam/openIdConnectProvider:OpenIdConnectProvider::main]
  | [2022-08-10T06:38:52Z]         [provider=urn:pulumi:dev-apse2-main::aws-eks::pulumi:providers:aws::dev-ap-southeast-2::bcae0a79-1844-4ceb-9bec-ed82206828a3]
  | [2022-08-10T06:38:52Z]       ~ tags: {
  | [2022-08-10T06:38:52Z]           + pulumi:Stack    : "dev-apse2-main"
  | [2022-08-10T06:38:52Z]         }
  | [20error: unable to validate AWS credentials - see https://pulumi.io/install/aws.html for details on configuration

Steps to reproduce

        docker-compose run --rm pulumi \
          sh -c 'pulumi preview --diff \
                   --debug \
                   --stack "${PULUMI_STACK}" \
                   --suppress-permalink=true \
                   --emoji \
                   --color always'

After attempting to upgrade from 5.6 -> 5.10.

This runs fine locally, where I have AWS credentials in my ~/.aws/credentials file, but fails on a CI runner that uses an instance profile. Have tried various iterations of the Provider parameters such as skipCredentialsValidation.

Expected Behavior

AWS credentials should be obtained.

Actual Behavior

Pulumi error.

[2022-08-10T06:45:44Z] debug: [aws-sdk-go-v2] request failed with unretryable error failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable  

Versions used

@aws-sdk/client-auto-scaling      3.142.0
@aws-sdk/client-ec2               3.142.0
@aws-sdk/client-eks               3.142.0
@aws-sdk/client-sts               3.142.0
@aws-sdk/credential-providers     3.142.0
@aws-sdk/types                    3.127.0
@elastic/elasticsearch            7.17.0
@kubernetes/client-node           0.17.0
@pulumi/aws                       5.10.0
@pulumi/awsx                      0.40.0
@pulumi/eks                       0.41.2
@pulumi/kubernetes                3.20.2
@pulumi/pulumi                    3.25.0
@types/js-yaml                    4.0.5
@types/uuid                       8.3.4
@typescript-eslint/parser         5.32.0
handlebars                        4.7.7
js-yaml                           4.1.0
retry-axios                       2.6.0
uuid                              8.3.2
@types/jest                       27.4.1
@types/node                       18.0.4
@typescript-eslint/eslint-plugin  5.32.0
axios                             0.27.2
eslint                            8.19.0
eslint-config-prettier            8.3.0
eslint-plugin-import              2.26.0
exponential-backoff               3.1.0
jest                              27.5.1
lint-staged                       12.5.0
prettier                          2.7.1
ts-jest                           27.1.3
typescript                        4.7.4

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rdanno]

I had a similar issue when doing inheritance because export AWS_PROFILE=account-x was being set. I removed this and it fixed the error.

[sushantkumar-amagi]

Can you try setting the aws:skipCredentialsValidation flag to True and aws:skipMetadataApiCheck to False

[simwak]

@sushantkumar-amagi Having the same issue even with both of the flags set.

[mwalker-g2u]

I am seeing something I think is similar. Trying to use an AWS KMS for the secret store from a GitHub action runner using OIDC to get credentials I get:

stderr: Command failed with exit code 255: pulumi stack select pulumi-oidc-test.dev --non-interactive
error: constructing secrets manager of type "cloud": secrets (code=Unknown): operation error KMS: Decrypt, failed to sign request: failed to retrieve credentials: no EC2 IMDS role found, operation error ec2imds: GetMetadata, failed to get API token, cannot get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed
 err?: Error: Command failed with exit code 255: pulumi stack select pulumi-oidc-test.dev --non-interactive
error: constructing secrets manager of type "cloud": secrets (code=Unknown): operation error KMS: Decrypt, failed to sign request: failed to retrieve credentials: no EC2 IMDS role found, operation error ec2imds: GetMetadata, failed to get API token, cannot get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed

If I use the AWS CLI to use the KMS to decrypt a secret it works fine. I don't have an AWS_PROFILE set. I have tried the aws:skipCredentialsValidation and aws:skipMetadataApiCheck settings as above.

I am using pulumi/Actions@v3 with version 3.40.2

[barryrobison]

Can you try setting the aws:skipCredentialsValidation flag to True and aws:skipMetadataApiCheck to False

This issue is fixed as of pulumi-aws 5.14 w/ the above settings, thanks!

[robotrapta]

I was able to fix this by running

pulumi config set aws:skipMetadataApiCheck false

as described in https://github.com/pulumi/pulumi-aws/issues/1692