AWS STS errors

[GustavoGama-DBACheck]

The AWS setup is a complete issue.

Expected behavior

Once the login is established and successful, the interaction with AWS API must be transparent for the developer. The user does not have temporary session, is a IAM user with access keys.

Current behavior

pulumi login s3://operations-tests-3 Logged in to Gustavos-MacBook-Pro.Home as gustavogama (s3://operations-tests-3) pulumi up Previewing update (dev-core-infra): Type Name Plan Info pulumi:pulumi:Stack core-infra-dev-core-infra 2 errors; 10 messages

Diagnostics: pulumi:pulumi:Stack (core-infra-dev-core-infra): Error: invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred: * error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired : Error: invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred: * error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired at Object.callback (/snapshot/awsx/node_modules/@pulumi/pulumi/runtime/invoke.js:148:33) at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client.ts:338:26) at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34) at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48) at /snapshot/awsx/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24 at processTicksAndRejections (node:internal/process/task_queues:78:11)

error: Error: invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred:
    * error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired


    at Object.callback (/snapshot/awsx/node_modules/@pulumi/pulumi/runtime/invoke.js:148:33)
    at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client.ts:338:26)
    at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
    at Object.onReceiveStatus (/snapshot/awsx/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
    at /snapshot/awsx/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
    at processTicksAndRejections (node:internal/process/task_queues:78:11)
error: Program failed with an unhandled exception:
Traceback (most recent call last):
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/resource.py", line 916, in do_rpc_call
    return monitor.RegisterResource(req)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred:
    * error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired

"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:62676 {grpc_message:"invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred:\n\t* error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired\n\n", grpc_status:2, created_time:"2023-06-11T15:42:58.999224+01:00"}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pulumi-language-python-exec", line 197, in <module>
    loop.run_until_complete(coro)
  File "/Users/gustavogama/.pyenv/versions/3.9.4/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 136, in run_in_stack
    await run_pulumi_func(lambda: Stack(func))
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
    await wait_for_rpcs()
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 120, in wait_for_rpcs
    raise exception
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/resource.py", line 1001, in do_register_resource_outputs
    serialized_props = await rpc.serialize_properties(outputs or {}, {})
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/rpc.py", line 208, in serialize_properties
    result = await serialize_property(
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/rpc.py", line 284, in serialize_property
    "urn": await serialize_property(
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/rpc.py", line 376, in serialize_property
    is_known = await output._is_known
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/rpc_manager.py", line 71, in rpc_wrapper
    result = await rpc
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/output.py", line 103, in is_value_known
    return await is_known and not contains_unknowns(await future)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/output.py", line 103, in is_value_known
    return await is_known and not contains_unknowns(await future)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/output.py", line 103, in is_value_known
    return await is_known and not contains_unknowns(await future)
  [Previous line repeated 29 more times]
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/resource.py", line 921, in do_register
    resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
  File "/Users/gustavogama/.pyenv/versions/3.9.4/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/resource.py", line 918, in do_rpc_call
    handle_grpc_error(exn)
  File "/Users/gustavogama/boost-it/aws_infra_v2/core-infra/venv/lib/python3.9/site-packages/pulumi/runtime/settings.py", line 273, in handle_grpc_error
    raise grpc_error_to_exception(exn)
Exception: invocation of aws:index/getAvailabilityZones:getAvailabilityZones returned an error: 1 error occurred:
    * error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 87fecadb-d8fc-43ac-8147-a6f49179ce6d, api error ExpiredToken: The security token included in the request is expired

Steps to reproduce

1. Create AWS
2. Create USER cicd with AdministratorAccess
3. Add user credentyials to .aws config and credentials files
4. Login into AWS or define env variables
5. Create BUCKET
6. pulumi login s3://<BUCKET NAME>
7. pulumi new with aws-python (make the region the same of the bucket)
8. c change main to use awsx to create a vpc
9. pulumi up -> and the result is in previous section
10. The issue is the sts:GetCallerIdentity

Context (Environment)

aws sts get-caller-identity --query "Account" --output text 882205788219

{ "UserId": "AIDA42Z4QBA533JXXXXX", "Account": "88220578XXXX", "Arn": "arn:aws:iam::88220578XXXX:user/cicd-user" } (END)

env virables: AWS_ACCESS_KEY_ID=XXXXXXX4QBA56JJXXXXX AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXJh4WbJ/ZIOTirOh4spYzpdXXXXX AWS_PROFILE=cicd-user AWS_REGION=eu-west-1

Affected feature

deploy infra into AWS

[danielrbradley]

Hi @GustavoGama-DBACheck sorry you've been having some challenges getting the AWS authentication configured.

The error message seems quite certain that the provider is finding an existing session token which it's using. This could perhaps be due to having AWS_SESSION_TOKEN set in your environment, in your AWS or Pulumi config, or set explicitly in your program.

Other than following the installation guide it's not obvious what other factors might be causing this behaviour. Perhaps another approach you could try is to set this up on a clean machine installation - perhaps via a docker container to see if there's something in your ambient environment causing the issue.

[t0yv0]

Closing as stale - unfortunately our team cannot make progress on this one without a repro. Please reopen with a repro if you are still hitting a problem.