`ERROR: Permission denied while accessing Artifact Registry.` despite service account having permission artifactregistry.repositories.get

[colinrsmall]

TL;DR

Deploying to gcloud run is failing with the error ERROR: (gcloud.run.deploy) PERMISSION_DENIED: Permission 'artifactregistry.repositories.get' denied on resource '...' (or it may not exist)., despite the service account I am authenticating with being granted the permission. The service account has the required roles from https://cloud.google.com/run/docs/deploying-source-code#permissions_required_to_deploy.

Expected behavior

No response

Observed behavior

No response

Action YAML

name: Test and Deploy

on:
  push:
    branches:
      - dev
      - prod

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install pytest pytest-cov flake8 mypy
      - name: flake8
        run: flake8 . --ignore=W605,E501,W503

  deploy_dev:
    if: github.ref == 'refs/heads/dev'
    needs: lint
    runs-on: ubuntu-latest
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
    - id: checkout_dev
      name: Checkout code
      uses: actions/checkout@v4

    - id: auth_dev
      name: Authenticate with Google Cloud
      uses: google-github-actions/auth@v2
      with:
        project_id: '...'
        workload_identity_provider: ${{ vars.WORKLOAD_IDENTITY_PROVIDER }}

    - id: deploy_dev
      name: Deploy to beta service
      uses: 'google-github-actions/deploy-cloudrun@v2'
      with:
        service: '${{ vars.BETA_SERVICE }}'
        region: 'us-central1'
        source: '.'

  deploy_prod:
    if: github.ref == 'refs/heads/prod'
    needs: lint
    runs-on: ubuntu-latest
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
    - id: checkout_prod
      name: Checkout code
      uses: actions/checkout@v2

    - id: auth_prod
      name: Authenticate with Google Cloud
      uses: google-github-actions/auth@v2
      with:
        workload_identity_provider: ${{ env.WORKLOAD_IDENTITY_PROVIDER }}
        service_account: ${{ env.SERVICE_ACCOUNT }}

    - id: deploy_prod
      name: Deploy to production service
      uses: 'google-github-actions/deploy-cloudrun@v2'
      with:
        service: '${{ vars.PROD_SERVICE }}'
        source: "."

Log output

##[debug]Caching tool gcloud 463.0.0 x64
##[debug]source dir: /home/runner/work/_temp/9480a076-9ef5-4a2a-91ec-5f4f335f40d8/google-cloud-sdk
##[debug]destination /opt/hostedtoolcache/gcloud/463.0.0/x64
##[debug]finished caching tool
##[debug]Running command: gcloud --quiet auth login --force --cred-file /home/runner/work/mv_backend/mv_backend/gha-creds-9f6a4748a1184e85.json
Successfully authenticated
Running: gcloud run deploy mv-backend-beta --quiet --source . --update-labels managed-by=github-actions,commit-sha=b360e992c5e859538227184e638fc67ef5cbf163 --platform managed --format json --region us-central1
Error: google-github-actions/deploy-cloudrun failed with: failed to execute gcloud command `gcloud run deploy mv-backend-beta --quiet --source . --update-labels managed-by=github-actions,commit-sha=b360e992c5e859538227184e638fc67ef5cbf163 --platform managed --format json --region us-central1`: ERROR: Permission denied while accessing Artifact Registry. Artifact Registry access is required to deploy from source.
ERROR: (gcloud.run.deploy) PERMISSION_DENIED: Permission 'artifactregistry.repositories.get' denied on resource '//artifactregistry.googleapis.com/projects/.../locations/us-central1/repositories/cloud-run-source-deploy' (or it may not exist).
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: artifactregistry.googleapis.com
  metadata:
    permission: artifactregistry.repositories.get
    resource: projects/.../locations/us-central1/repositories/cloud-run-source-deploy
  reason: IAM_PERMISSION_DENIED
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Deploy to beta service

Additional information

Happy to provide the runner's debug output. Are there any sensitive values in the debug output that need to be redacted?

[ranil-bee]

We are seeing a similar issue and was going to raise a ticket. In our case, even though the build fails - the deployment still succeeds. This issue only started a couple of weeks ago,

[colinrsmall]

In our case, even though the build fails - the deployment still succeeds.

If the deployment succeeds but the build fails, what artifact is your service running off of? Or do you mean to say that the build failing is a false positive?

[ranil-bee]

In our case, even though the build fails - the deployment still succeeds.

If the deployment succeeds but the build fails, what artifact is your service running off of? Or do you mean to say that the build failing is a false positive?

In our case, the build failing is a false positive.

So we build and push our image to artifactory and the deployment deploys that labelled image. So the push stage works, and we can see the new images getting deployed too (we can see the new image deployed on cloud run), but we get an error saying permission denied. I have manually validated that the service account can run that command without any errors.

Run google-github-actions/deploy-cloudrun@v2 /usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/b91e8f3c-cbd8-4a9c-a459-3a478049b482 -f /home/runner/work/_temp/4c7c986d-8b36-496d-afad-0363a022f9f5 Successfully authenticated Running: gcloud run deploy casiax-bifrost-beta --quiet --image australia-southeast1-docker.pkg.dev/french-lavender/casiax-images/casiax-bifrost:main --platform managed --format json --region australia-southeast1 --project french-lavender --ingress internal Error: google-github-actions/deploy-cloudrun failed with: failed to execute gcloud command gcloud run deploy casiax-bifrost-beta --quiet --image australia-southeast1-docker.pkg.dev/french-lavender/casiax-images/casiax-bifrost:main --platform managed --format json --region australia-southeast1 --project french-lavender --ingress internal: Deploying container to Cloud Run service [casiax-bifrost-beta] in project [french-lavender] region [australia-southeast1] Deploying... Creating Revision.................failed Deployment failed ERROR: (gcloud.run.deploy) PERMISSION_DENIED: Permission 'run.operations.get' denied on resource '' (or resource may not exist).

Update 1 So digging further the resource `projects/french-lavender/locations/australia-southeast1/operations/b5a52ce9-f623-4654-96de-c07e819c3545' does not exist. It clearly identifies the project, region and service that is updating above - so not sure what this operations resource is used for.

Update 2 Don't understand how but there does appear to be a permissions issue difference when run from Github workflow and manually (with the service account). I created a fully permissioned service account and it ran without an error. Will work on a service account with the appropriate level of permissions but please ignore these error for now.

[sethvargo]

Is this related to https://issuetracker.google.com/issues/322167526?

[ranil-bee]

Is this related to https://issuetracker.google.com/issues/322167526?

Thanks this answers and explains most of my mystery. Does it help address the original ticket that you raised?

[colinrsmall]

Was just going to respond that it looks more related to @ranil-bee's issue. Unlike users on the Google issue tracker, I can't seem to reproduce my issue running gcloud run deploy ... --service-account={service_account} locally, perhaps because I am deploying my code from source (using Google Cloud buildpacks)?

If nothing else, I'm having trouble squaring the error message I'm seeing with what the policy troubleshooter reports. The only thing I can think of is that the artifactregistry.repositories.get permission is not making it to the token produced by authentication and thus the identity created by the WIF pool that is impersonating the service account doesn't have it (which I guess makes this a problem for the authentication repo and not the cloud run deploy repo?). I'm going to try deploying from an existing image and potentially using a service account key instead.

[colinrsmall]

I can confirm that deploying from source works when I authenticate the service account with a service account key instead of WIF (WIF still doesn't work as of this comment). @sethvargo should I open an issue in the authenticate script repo?

[sethvargo]

Sure, please include the debug logs and IAM permissions that you've granted to the WIF pool though.