Workflow identity federation with managed identity in Azure Kubernetes Service

[gunndabad]

We're looking at using managed identity in AKS and identity federation to allow our apps in AKS to access resources in Google and I'm trying to work out the best way of creating a GoogleCredential.

Using a JSON config file with a url credential_source and the http://169.254.169.254/metadata/identity/oauth2/token doesn't work for AKS; we have to get a token as shown here instead then pass that to the STS endpoint then create another token from that.

I have this working by doing all the HTTP requests manually then passing the result to GoogleCredential.FromAccessToken() but this feels like a lot of work when ExternalAccountCredential seems to already know how to do the STS parts.

Questions I have:

1. Is this the best way to achieve this?
2. Is there a way to change the access token for a GoogleCredential after it's been initialized and if not what's the recommend approach for refreshing the credential?
3. Is there a way to create a custom ExternalAccountCredential ourselves that does the initial token request (it looks like everything we'd need to extend is internal)?

[amanda-tarafa]

1. Is this the best way to achieve this?

We don't support this type of credential out of the box, no. So any solution will require some work on your side. I will bring this up internally to evaluate options, but just for expectations, making a decision here on whether we want to support this credential out of the box or not may take time, as we usually want to have feature parity across Auth libraries.

2. Is there a way to change the access token for a GoogleCredential after it's been initialized and if not what's the recommend approach for refreshing the credential?

Not from calling code, no. You can extend ServiceCredential and implement RequestAccessTokenAsync, which will only be called when the existing token is about to expire. But see my next answer first.

3. Is there a way to create a custom ExternalAccountCredential ourselves that does the initial token request (it looks like everything we'd need to extend is internal)?

We are usually cautious when opening up classes for extension. But that doesn't mean we won't consider it. If I understand this correctly, STS exchange, the possibility of impersonation, etc. it's all the same. You just need to obtaing the subject token differently, right? So, you should be able to extend ExternalAccountCredential and implement GetSubjectTokenAsyncImpl? The only thing we'd need to make protected is the Initializer? Is that right? Then you can extend the initializer to add the specifics of these credentials, like the token path and tenant ID (we already have Client ID) and use those in your implementation of GetSubjectTokenAsyncImpl. And if you need your credential to be wrapped by a GoogleCredential I would consider adding a GoogleCredential.FromExternalAccountCredential method. Let me know if you think this would work, or if I'm missing something.

[gunndabad]

Thanks

You just need to obtaing the subject token differently, right? So, you should be able to extend ExternalAccountCredential and implement GetSubjectTokenAsyncImpl? The only thing we'd need to make protected is the Initializer? Is that right?

If the internal constructor and WithoutImpersonationConfigurationImpl method were made protected too I think that would work.

[amanda-tarafa]

Yes, the constructor of course, and WithoutImpersonationConfigurationImpl as well, it's absract after all. I might make some changes to this last one so that it's more explicit on what implementers have to do.

I'm marking this as a feature request to open up ExternalAccountCredential for extensions.

[amanda-tarafa]

@gunndabad I have a question for you. The token from here that you send to the STS endpoint, i.e. the subject token, is it the AccessToken obtained via either GetTokenAsync or GetToken or is it the JWT obtained via ReadJWTFromFS. I'm guessing is the former, but just want to make sure.

[gunndabad]

@amanda-tarafa yes it’s the former.

[amanda-tarafa]

@gunndabad I've been talking to folks in the Identity team about this and they suggested you follow instructions to Configure workload identity federation in AKS, in particular, that you enable the OIDC issuer feature and use file sourced credentials. Things should work out of the box from there.

Regardless, there's an effort across Google Auth libraries in all langauges to open up external credentials for extensibility, so we'll work on that. It just might take a little while longer, as we want to be aligned in requirements etc.

Do let me know if enabling the OIDC issuer feature works for you, so I know what the urgency on implementing extensibility for externals credentials is.

[gunndabad]

@amanda-tarafa we've set that up already (largely following the guide at https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster) but couldn't get file-sourced credentials to work.

[amanda-tarafa]

OK, just to report back to the Indentity team, when you say you couldn't get file-sourced credentials to work, what was the problem?

• Was the credential not being found at all?
• Whas there some error at some of the steps involved in obtaining the token?
• A token was obtained but it was rejected by Google?

If you have a stack trace for whatever error that'd be useful as well.

I'll make an attempt to try this myself and see where it gets me.

[gunndabad]

I've tried this config:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/<our project ID>/locations/global/workloadIdentityPools/<our pool ID>/providers/<our provider ID>",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<our service account>@get-an-identity.iam.gserviceaccount.com:generateAccessToken",
  "credential_source": {
    "url": "https://login.microsoftonline.com/<our tenant ID>/oauth2/v2.0/token?api-version=2018-02-01&resource=api://a1039720-f1aa-4d81-9996-b305299bf0ce",
    "headers": {
      "Metadata": "True"
    },
    "format": {
      "type": "json",
      "subject_token_field_name": "access_token"
    }
  }
}

With that I get:

Unhandled exception. Google.Apis.Auth.OAuth2.SubjectTokenException: An error occurred while attempting to obtain the subject token for UrlSourcedExternalAccountCredential
 ---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Google.Apis.Json.NewtonsoftJsonSerializer.Deserialize(String input, Type type)
   at Google.Apis.Json.NewtonsoftJsonSerializer.Deserialize[T](String input)
   at Google.Apis.Auth.OAuth2.UrlSourcedExternalAccountCredential.GetSubjectTokenAsyncImpl(CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.ExternalAccountCredential.GetSubjectTokenAsync(CancellationToken taskCancellationTokne)
   --- End of inner exception stack trace ---
   at Google.Apis.Auth.OAuth2.ExternalAccountCredential.GetSubjectTokenAsync(CancellationToken taskCancellationTokne)
   at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestStsAccessTokenAsync(CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.RefreshTokenAsync()
   at Google.Apis.Auth.TaskExtensions.<>c__DisplayClass0_0`1.<<WithCancellationToken>g__ImplAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String authUri, CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.ServiceCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Google.Apis.Http.ConfigurableMessageHandler.CredentialInterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Google.Apis.Auth.OAuth2.Requests.RequestExtensions.PostJsonAsync(Object request, HttpClient httpClient, String url, CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.Requests.RequestExtensions.PostJsonAsync(Object request, HttpClient httpClient, String url, IClock clock, ILogger logger, CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.ImpersonatedCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.ExternalAccountCredential.RequestAccessTokenAsync(CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.RefreshTokenAsync()
   at Google.Apis.Auth.TaskExtensions.<>c__DisplayClass0_0`1.<<WithCancellationToken>g__ImplAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String authUri, CancellationToken cancellationToken)
   at Google.Apis.Auth.OAuth2.ServiceCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Google.Apis.Http.ConfigurableMessageHandler.CredentialInterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken)
   at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken)
   at Google.Cloud.BigQuery.V2.BigQueryClientImpl.GetDatasetAsync(DatasetReference datasetReference, GetDatasetOptions options, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in /home/app/app/Program.cs:line 14
   at Program.<Main>(String[] args)

[jskeet]

At a guess, that means that Microsoft is returning HTML rather than JSON... that's normally what "I was trying to parse JSON but I got < at line 0 position 0" means.

[gunndabad]

I suspect the MS token request is failing as there's no way to provide the contents of the AZURE_FEDERATED_TOKEN_FILE environment variable.

[amanda-tarafa]

Thanks for the info. With this, I can look into a few things:

• If the subject token request is failing, why do we attempt to parse the response as JSON?
• If there are changes to your credential configuration that can be made to make this work. The Identity team did say to use file-sourced credential for this case. What I understood from my conversation with them is that when you enable the OIDC issuer feature, then you can use the token in AZURE_FEDERATED_TOKEN_FILE directly, so you'd put that path on the file-sourced credential configuration. I'll double check all of this.

[gunndabad]

can use the token in AZURE_FEDERATED_TOKEN_FILE directly, so you'd put that path on the file-sourced credential configuration

Using that token directly with the https://sts.googleapis.com/v1/token endpoint doesn't work; it first needs to be exchanged for another access token using the MS token endpoint https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token

[amanda-tarafa]

OK, I'll get back with this info to the Identity team, just to at least clarify what they meant.

It's likely then that this will fall into the "let's open external credential for extension" as we had been talking about before. I'll start planning for that as well.