Cannot refresh credentials to retrieve an ID token

[GergelyKalmar]

TL;DR

I'm trying to get an ID token to use for authenticating towards Google Cloud Run services in GitHub Actions in Python code, however, for some reason it does not seem to be working.

Expected behavior

Receive the ID token.

Observed behavior

I got the following error:

    ...
    credentials.refresh(request=Request())  # type: ignore[no-untyped-call]
.../lib/python3.8/site-packages/google/auth/external_account.py:397: in refresh
    self._impersonated_credentials.refresh(request)
.../lib/python3.8/site-packages/google/auth/impersonated_credentials.py:250: in refresh
    self._update_token(request)
.../lib/python3.8/site-packages/google/auth/impersonated_credentials.py:282: in _update_token
    self.token, self.expiry = _make_iam_token_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

request = <google.auth.transport.requests.Request object at ...>
principal = '[email protected]'
headers = {'Content-Type': 'application/json', 'authorization': '*** 'x-allowed-locations': '0x0', 'x-goog-api-client': 'gl-python/3.8.18 auth/2.29.0 auth-request-type/at cred-type/imp'}
body = b'{"delegates": null, "scope": null, "lifetime": "3600s"}'
iam_endpoint_override = 'https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken'

    def _make_iam_token_request(
        request, principal, headers, body, iam_endpoint_override=None
    ):
        """Makes a request to the Google Cloud IAM service for an access token.
        Args:
            request (Request): The Request object to use.
            principal (str): The principal to request an access token for.
            headers (Mapping[str, str]): Map of headers to transmit.
            body (Mapping[str, str]): JSON Payload body for the iamcredentials
                API call.
            iam_endpoint_override (Optiona[str]): The full IAM endpoint override
                with the target_principal embedded. This is useful when supporting
                impersonation with regional endpoints.
    
        Raises:
            google.auth.exceptions.TransportError: Raised if there is an underlying
                HTTP connection error
            google.auth.exceptions.RefreshError: Raised if the impersonated
                credentials are not available.  Common reasons are
                `iamcredentials.googleapis.com` is not enabled or the
                `Service Account Token Creator` is not assigned
        """
        iam_endpoint = iam_endpoint_override or _IAM_ENDPOINT.format(principal)
    
        body = json.dumps(body).encode("utf-8")
    
        response = request(url=iam_endpoint, method="POST", headers=headers, body=body)
    
        # support both string and bytes type response.data
        response_body = (
            response.data.decode("utf-8")
            if hasattr(response.data, "decode")
            else response.data
        )
    
        if response.status != http_client.OK:
>           raise exceptions.RefreshError(_REFRESH_ERROR, response_body)
E           google.auth.exceptions.RefreshError: ('Unable to acquire impersonated credentials', '{\n  "error": {\n    "code": 400,\n    "message": "Request contains an invalid argument.",\n    "status": "INVALID_ARGUMENT"\n  }\n}\n')

.../lib/python3.8/site-packages/google/auth/impersonated_credentials.py:100: RefreshError

Action YAML

See https://github.com/logikal-io/github-workflows/blob/main/.github/workflows/run-python-tests.yml

It is essentially:

- name: Authenticate to Google Cloud Platform
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ inputs.gcp-testing-workload-identity-provider }}
          service_account: ${{ inputs.gcp-testing-service-account }}

- name: Run pytest
        run: orb --command 'pytest ${{ inputs.pytest-options }}'

Log output

------------------------------ Captured log call -------------------------------
2024-09-12 17:06:04.630 DEBUG Loading default credentials (stormware.google.auth:91)
2024-09-12 17:06:04.630 DEBUG Checking /home/runner/work/.../gha-creds-....json for explicit credentials as part of auth process... (google.auth._default:255)
2024-09-12 17:06:04.631 DEBUG Making request: GET https://pipelinesghubeus7.actions.githubusercontent.com/.../_apis/distributedtask/hubs/Actions/plans.../jobs/.../idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2Fprojects%2...%2Flocations%2Fglobal%2FworkloadIdentityPools%2Fci-cd%2Fproviders%2Fgithub-actions (google.auth.transport.requests:185)
2024-09-12 17:06:04.920 DEBUG Making request: POST https://sts.googleapis.com/v1/token (google.auth.transport.requests:185)
2024-09-12 17:06:04.977 DEBUG Making request: POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken (google.auth.transport.requests:185)

Additional information

The following code snippet was triggering the error:

from google.auth import default
from google.auth.transport.requests import Request

credentials = default()[0]
credentials.refresh(request=Request())  # <-- here
print(credentials.id_token)

I understand that there is a way to generate an ID token as an output value, however, injecting that value into the Python script seems like a hack as opposed to using the default credential flow. Shouldn't this approach work too?

[github-actions[bot]]

Hi there @GergelyKalmar :wave:!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

[sethvargo]

This seems like a bug with the python library. Which library are you using and what version is it?

[GergelyKalmar]

We're using google-auth==2.29.0. It seems an id_token related bug was fixed in 2.30.0, let me try to update to that and see if it makes any difference.

[GergelyKalmar]

Ok, I'm blocked releasing with the new version by https://github.com/googleapis/google-auth-library-python/issues/1593, so we'll need to wait that out.

[sethvargo]

Hmm okay - I'm not a python expert, so I'm not sure how to help debug that. Let's keep an eye on it, and see what the team says.

[sethvargo]

Hi @GergelyKalmar - any update?

[GergelyKalmar]

No, the blocking issue is still open. Seems like the guys on that library are not very quick at fixing CI/CD issues :). If you can reach out to them internally and nudge them a bit, that might help!

[GergelyKalmar]

Okay, I've updated the library, it still throws the same error. I also tried to do token = oauth2.id_token.fetch_id_token(...) instead, that was complaining about google.auth.exceptions.DefaultCredentialsError: Neither metadata server or valid service account credentials are found.. In general I'd like my tests to trigger another Cloud Run service for integration tests, however, I can't seem to be able to generate a token that allows me to do that in any way.

[GergelyKalmar]

I suppose the best option then would be to use this action to generate the ID token and then inject it into the environment. I'd have one question regarding this: if I choose token_format = 'id_token', does the action still generate the usual credentials in addition to the ID token, or do I only get the ID token? Asked differently: do I need to call this action twice if I need both the regular credentials and the ID token?

[sethvargo]

You'll always get an auth_token, but you'll get one of access_token, id_token, or nothing depending on token_format.

[GergelyKalmar]

I should have been more precise in my question: can I use workload identity federation and token_format in the same step, or will I need to use the action twice (once for WIF and once to get the id_token)?

[sethvargo]

Hmm that's actually a good question. I think so, but they best thing to do would be to try it.

[fengkai-tink]

Updating to google-auth 2.38.0 doesn't help

[tabrezm]

In my case, after enabling IAM and STS logs, I saw this extra detail:

message: "Scope required. "

Try this instead:

    request = requests.Request()
    credentials = credentials.with_scopes(
        ["https://www.googleapis.com/auth/cloud-platform"]
    )
    credentials.refresh(request)

[sethvargo]

#487 #488