Running `pulumi up` sends an incorrect subject identifier to Providers

[toriancrane]

What happened?

Related to this issue.

After configuring Pulumi ESC to work with Azure and Pulumi IaC, I tried to run the pulumi up command to have my Azure resources deployed and ran into the following error:

  azure-native:resources:ResourceGroup (resource_group):
    error: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS700213: No matching federated identity record found for presented assertion subject 'pulumi:environments:org:pulumi:env:<yaml>'. Please check your federated identity credential Subject, Audience and Issuer against the presented assertion. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation Trace ID: 20f6c59a-9821-47da-afb7-aafa724e6b01 Correlation ID: 1b19f0d7-cba3-4b0a-ae44-7489adf2454b Timestamp: 2023-11-07 09:11:45Z","error_codes":[700213],"timestamp":"2023-11-07 09:11:45Z","trace_id":"20f6c59a-9821-47da-afb7-aafa724e6b01","correlation_id":"1b19f0d7-cba3-4b0a-ae44-7489adf2454b"}

Somewhere in this OIDC workflow, pulumi:environments:org:pulumi:env:<yaml> is being sent as the subject identifier rather than the one that includes the name of my ESC environment.

Example

1. Configure Azure OIDC with Pulumi ESC.
2. Add the following definition to the environment file:

values:
  azure:
    login:
      fn::open::azure-login:
        clientId: your-client-id
        tenantId: you-tenant-id
        subscriptionId: your-subscription-id # without the `/subscriptions/` prefix
        oidc: true
  environmentVariables:
    ARM_USE_OIDC: 'true'
    ARM_CLIENT_ID: ${azure.login.clientId}
    ARM_TENANT_ID: ${azure.login.tenantId}
    ARM_OIDC_REQUEST_TOKEN: ${azure.login.oidc.token}
    ARM_OIDC_TOKEN: ${azure.login.oidc.token}
    ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
    ARM_OIDC_REQUEST_URL: https://api.pulumi.com/oidc

3. Create a new Pulumi Python project (pulumi new azure-python)
4. Update the stack settings file to import ESC environment file:

environment:
  - your-environment-name

5. Run pulumi up -y

Output of pulumi about

CLI          
Version      3.92.0
Go Version   go1.21.3
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  2.15.0
python        unknown

Host     
OS       amazon
Version  2
Arch     x86_64

This project is written in python: executable='/home/ec2-user/.pyenv/shims/python3' version='3.8.12'

Current Stack: pulumi/az-pulumi/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/v-torian-pulumi-corp
User           v-torian-pulumi-corp
Organizations  v-torian-pulumi-corp, zephyr, pulumi
Token type     personal

Dependencies:
NAME                 VERSION
pip                  23.3.1
pulumi-azure-native  2.15.0
setuptools           68.2.2
wheel                0.41.3

Pulumi locates its logs in /tmp by default

Additional context

When I add pulumi:environments:org:pulumi:env:<yaml> as the subject identifier to Azure, the workflow deploys successfully.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[pgavlin]

This is semi-intentional.

The reason that <yaml> is in there is that we play a bit of a trick with stack environments. In order to maintain the same sort of merging behavior we get with normal ESC environments, the list of environments for a stack is internally converted to an anonymous environment definition prior to opening. All anonymous environments have the name yaml in the subject identifier. We can't avoid the anonymous environment in the case that there are multiple environments in the stack's list, but we could do so if there is only a single environment. Not sure if that would be more or less confusing.

So in essence this is working as intended, though I agree that it's surprising when there's only a single listed environment.

[MitchellGerdisch]

I'm not sure I totally follow this, but it seems like the <yaml> bit is happening in Pulumi to act as the name for merged environments when a stack config references multiple environments. However, I would assert that Pulumi should track the name of the environment containing the OIDC declaration and use that when creating the subject. This way the user can properly manage the permissions related to the given OIDC environment and what it is allowed to do in Azure.

[JonParton]

Although this issue mentions azure Specifically I have just encountered this same issue for AWS also.

Might be worth expanding the subject to say this is a problem on all providers? I expect fixing one will fix them all? Or do you want separate issues?

We may also want to add the same caveat to the AWS OIDC Docs until resolved:

From my understanding of the intent of using RBAC with Teams to control access to Environments I agree with @MitchellGerdisch - Feels like the Environment sent in the OIDC Subject claim should be equal to the source environment that the OIDC credentials are declared in (Even when merged / imported into other environments). This would then allow us to be able to ensure the OIDC token can only be used by people / tokens that have been given access to that source environment on pulumi cloud!

For now I am going to have to allow the :<yaml> subject too but this then makes it unsecure as any environment within the organisation could then just reference these credentials and they can't be tied down!

@pierskarsenbarg - Just giving a reference that we have encountered this issue also if there is any help you can input in getting to a resolution 🙂

[desteves]

I hit this issue last night with GCP as the Provider.

To get GCP + ESC to work I added IAM principals with the Workload Identity User role as shown below yet did not define any Subject Claims.

For a Pulumi stack config yaml file: principal://iam.googleapis.com/projects/123123123123/locations/global/workloadIdentityPools/mypoolid/subject/pulumi:environments:org:myorgname:env:<yaml>

For the Pulumi Cloud ESC editor: principal://iam.googleapis.com/projects/123123123123/locations/global/workloadIdentityPools/mypoolid/subject/pulumi:environments:org:myorgname:env:myescenv

(I replaced actual values with dummies)

[arunkumar611]

We have added support for subjectAttributes to allow users to customize the OIDC token claims. We suggest using that to prevent having in the value of the subject.

See our docs for details on Custom Claims.

We also have examples for AWS, Azure, and GCP