Have TokenFilePath configurable via an environment variable, when using WorkloadIdentity

[mjnovice]

Describe the current behavior A description of how things are today. Currently the TokenFilePath is hardcoded to a fixed value: /var/run/secrets/tokens/azure-identity This gets problematic when using azure workload identity webhook in conjunction on a vcluster.

Describe the improvement How should things be changed or improved? It should be configurable via the some env var

Additional context Add any other context about the suggested improvement.

[matthchr]

Are you talking about this vcluster?

Can you describe a bit more what token ends up available at the hardcoded path of /var/run/secrets/tokens/azure-identity in the vcluster context? Is the problem here that there are 2 workload identity webhooks and both inject something into the ASO pod?

We're not necessarily opposed to making this configurable, but want to understand more about the scenario first.

[mjnovice]

@matthchr Hey Matthew, yes, that is the vcluster we are using.

If we use the --sync-labels options of the workload identity https://github.com/Azure/azure-workload-identity running along with the label azure.workload.identity/use="true" on the host AKS cluster, it adds /var/run/secrets/tokens/azure-identity-token as the path where the token is added.

[mjnovice]

@matthchr let me know, would be happy to work on this, if there is consensus.

[matthchr]

I assume you mean the --sync-labels option of vcluster? I don't think there's a workload identity option of that name.

You don't need to set azure.workload.identity/use=true for ASO to use workload identity. In fact you don't need to run the workload identity webhook at all. See the docs.

I don't think that setting azure.workload.identity/use="true" on the ASO pod will break anything though. It will cause the webhook to trigger, but the webhook will not overwrite existing env variables.

ASO already has hardcoded injection of the token for its service account at /var/run/secrets/tokens/azure-identity, the standard workload identity path injects at /var/run/secrets/azure/tokens/azure-identity-token. I think if you annotate the ASO ServiceAccount, you'll see two injected volumes, one which is the hardcoded one, and one which is the one injected from the workload identity webhook. The one injected from the workload identity webhook will be ignored but the other one should always work, assuming you've followed the setup instructions and created a FederatedIdentityCrednetial like so:

az identity federated-credential create --name aso-federated-credential --identity-name ${MI_NAME} --resource-group ${MI_RESOURCE_GROUP} --issuer ${SERVICE_ACCOUNT_ISSUER} --subject "system:serviceaccount:azureserviceoperator-system:azureserviceoperator-default" --audiences "api://AzureADTokenExchange"

and have set that MI CLIENT ID as the AZURE_CLIENT_ID when installing ASO.

What ServiceAccount owns the token being injected at /var/run/secrets/tokens/azure-identity-token in your scenario? Is it the ASO service account?

Can you share what your ASO Deployment looks like after the workload identity webhook has run? This is what a standard workload-identity enabled ASO would look like:

spec:
  ... (elided)
  template:
    ... (elided)
    spec:
      containers:
      - args:
        - --metrics-addr=:8443
        - --secure-metrics=true
        - --profiling-metrics=false
        - --health-addr=:8081
        - --enable-leader-election
        - --v=2
        - --crd-pattern=*
        - --webhook-port=9443
        - --webhook-cert-dir=/tmp/k8s-webhook-server/serving-certs
        env:
        - name: AZURE_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: AZURE_CLIENT_ID
              name: aso-controller-settings
        - name: AZURE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: AZURE_CLIENT_SECRET
              name: aso-controller-settings
              optional: true
        - name: AZURE_TENANT_ID
          valueFrom:
            secretKeyRef:
              key: AZURE_TENANT_ID
              name: aso-controller-settings
        - name: AZURE_SUBSCRIPTION_ID
          valueFrom:
            secretKeyRef:
              key: AZURE_SUBSCRIPTION_ID
              name: aso-controller-settings
        - name: AZURE_CLIENT_CERTIFICATE
          valueFrom:
            secretKeyRef:
              key: AZURE_CLIENT_CERTIFICATE
              name: aso-controller-settings
              optional: true
        - name: AZURE_CLIENT_CERTIFICATE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: AZURE_CLIENT_CERTIFICATE_PASSWORD
              name: aso-controller-settings
              optional: true
        - name: AZURE_AUTHORITY_HOST
          valueFrom:
            secretKeyRef:
              key: AZURE_AUTHORITY_HOST
              name: aso-controller-settings
              optional: true
        - name: AZURE_RESOURCE_MANAGER_ENDPOINT
          valueFrom:
            secretKeyRef:
              key: AZURE_RESOURCE_MANAGER_ENDPOINT
              name: aso-controller-settings
              optional: true
        - name: AZURE_RESOURCE_MANAGER_AUDIENCE
          valueFrom:
            secretKeyRef:
              key: AZURE_RESOURCE_MANAGER_AUDIENCE
              name: aso-controller-settings
              optional: true
        - name: AZURE_TARGET_NAMESPACES
          valueFrom:
            secretKeyRef:
              key: AZURE_TARGET_NAMESPACES
              name: aso-controller-settings
              optional: true
        - name: AZURE_OPERATOR_MODE
          valueFrom:
            secretKeyRef:
              key: AZURE_OPERATOR_MODE
              name: aso-controller-settings
              optional: true
        - name: AZURE_SYNC_PERIOD
          valueFrom:
            secretKeyRef:
              key: AZURE_SYNC_PERIOD
              name: aso-controller-settings
              optional: true
        - name: USE_WORKLOAD_IDENTITY_AUTH
          valueFrom:
            secretKeyRef:
              key: USE_WORKLOAD_IDENTITY_AUTH
              name: aso-controller-settings
              optional: true
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        name: manager
        ... (elided)
        volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: azure-identity
          readOnly: true
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
      serviceAccount: azureserviceoperator-default
      serviceAccountName: azureserviceoperator-default
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: webhook-server-cert
      - name: azure-identity
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: api://AzureADTokenExchange
              expirationSeconds: 3600
              path: azure-identity

[mjnovice]

Hey @mattchr the problem starts with vcluster, when the volume is produced on the host cluster, which uses downward API and would look something like this

      - downwardAPI:
          items:
          - fieldRef:
....

And the above is not really updated by kubernetes.

You can try doing this on a vcluster.

[matthchr]

What are you actually trying to do though? Set one identity for all vclusters, without the user of the vcluster needing to go through the regular ASO installation process?

I was thinking that with vcluster the user would install ASO into the vcluster just as if it were another cluster. Assuming that's what they're doing, I don't follow why installation into a vcluster would be any different than installation into a standard cluster.

With that said, when I get some time I will try it out and see if I can suss out what isn't working.

[theunrepentantgeek]

We haven't fully understood this scenario and didn't manage to test it out ourselves. Please feel free to reopen if you want.