Reconsider the default aws:skipMetadataCheck setting

[jaxxstorm]

In #1288 we disabled the metadata check in order to speed Pulumi up.

Unfortunately, this breaks Pulumi when running with an EC2 instance role, meaning lots of scenarios like self hosted CI/CD runners will not work.

It's possible of course to fix this by setting the environment variable AWS_EC2_METADATA_DISABLED=false or pulumi config set aws:skipMetadataApiCheck false, but the it's extremely difficult to figure that out unless you inherently know about this setting. This default setting seems to be violating the principal of least surprise and is causing a headache for those users who are setting Pulumi up in CI/CD.

To rectify this, we need to either:

1. Provide concrete information back to the user about how to fix this. This might involve intercepting the AWS SDK calls and injecting information about why they're getting this information
2. Consider setting the default value back to false

it might be possible to intercept the error message, which currently looks a bit like this:

Error: NoCredentialProviders: no valid providers in chain
    caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
    SharedCredsLoad: failed to load profile, .
    EC2RoleRequestError: no EC2 instance role found
    caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env var

But this comes from the AWS SDK, so we'd need to trap the error

[SharpEdgeMarshall]

ATM is not possible to override through ENV VAR, for some reason setting AWS_EC2_METADATA_DISABLED=false doesn't work.

We had to change all our Providers and add aws:skipMetadataApiCheck false

https://github.com/pulumi/pulumi-aws/issues/1636

[seunggs]

Hi @SharpEdgeMarshall - I'm running into this issue as I'm trying to implement Pulumi Automation API. Is the only solution then to create an explicit AWS provider with { skipMetadataApiCheck: false } in the CustomResourcesOptions? Or is there a simpler solution?

[SharpEdgeMarshall]

@seunggs there are no simpler solutions that i know atm sorry.

[seunggs]

No worries - I just misunderstood the solution. I didn't know you meant adding skipMetadataApiCheck: false as Pulumi config. That's a simple solution! :) Thanks for sharing this.

[SharpEdgeMarshall]

Pay attention that if you put it in the config it works only for the default provider.

[seunggs]

Oh didn't know that - thanks for pointing that out.

[blarghmatey]

This is still a problem. It would be great if we can get the default changed back, or at least fix the fact that the environment variable isn't being picked up properly by the TF provider at the bottom of the chain.

[ringods]

This will at least make it more visible from the AWS Classic registry page: https://github.com/pulumi/registry/pull/1313

[jf]

for the record, I also got hit with needing to set aws:skipCredentialsValidation to true in order for my instance profile to work. Doh...

[robotrapta]

AFAICT this problem manifests with the unhelpful error message

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable

Dropping this here to help lost people like previous-me looking for an answer. The answer is:

pulumi config set aws:skipMetadataApiCheck false