auth Error: google-github-actions/auth failed with: retry function failed after 1 attempt: error message: Cannot read properties of undefined (reading 'message')

TL;DR

I have been seeing random failures in our Github Action auth step with the following error message:

Error: google-github-actions/auth failed with: retry function failed after 1 attempt: error message: Cannot read properties of undefined (reading 'message')

The action works 99% of the time, with the odd run showing this error here and there.

Example failed build Workflow file

Expected behavior

No response

Observed behavior

No response

Action YAML

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: tests
on:
  pull_request:
  push:
    branches:
      - main
  pull_request_target:
    types: [labeled]
  schedule:
  - cron:  '0 2 * * *'

jobs:
  integration:
    # run job on proper workflow event triggers (skip job for pull_request event from forks and only run pull_request_target for "tests: run" label)
    if: "${{ (github.event.action != 'labeled' && github.event.pull_request.head.repo.full_name == github.event.pull_request.base.repo.full_name) || github.event.label.name == 'tests: run' }}"
    name: integration tests
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [macos-latest, windows-latest, ubuntu-latest]
      fail-fast: false
    permissions:
      contents: 'read'
      id-token: 'write'
    steps:
      - name: Remove PR label
        if: "${{ github.event.action == 'labeled' && github.event.label.name == 'tests: run' }}"
        uses: actions/github-script@v6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            try {
              await github.rest.issues.removeLabel({
                name: 'tests: run',
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.payload.pull_request.number
              });
            } catch (e) {
              console.log('Failed to remove label. Another job may have already removed it!');
            }

      - name: Checkout code
        uses: 'actions/checkout@v3'
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          repository: ${{ github.event.pull_request.head.repo.full_name }}

      - name: Setup Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.18

      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/[email protected]'
        with:
          workload_identity_provider: ${{ secrets.PROVIDER_NAME }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}
          access_token_lifetime: 600s
      
      - id: 'secrets'
        name: Get secrets
        uses: 'google-github-actions/[email protected]'
        with:
          secrets: |-
            MYSQL_CONNECTION_NAME:${{ secrets.GOOGLE_CLOUD_PROJECT }}/MYSQL_CONNECTION_NAME
            MYSQL_USER:${{ secrets.GOOGLE_CLOUD_PROJECT }}/MYSQL_USER
            MYSQL_PASS:${{ secrets.GOOGLE_CLOUD_PROJECT }}/MYSQL_PASS
            MYSQL_DB:${{ secrets.GOOGLE_CLOUD_PROJECT }}/MYSQL_DB
            POSTGRES_CONNECTION_NAME:${{ secrets.GOOGLE_CLOUD_PROJECT }}/POSTGRES_CONNECTION_NAME
            POSTGRES_USER:${{ secrets.GOOGLE_CLOUD_PROJECT }}/POSTGRES_USER
            POSTGRES_USER_IAM:${{ secrets.GOOGLE_CLOUD_PROJECT }}/POSTGRES_USER_IAM_GO
            POSTGRES_PASS:${{ secrets.GOOGLE_CLOUD_PROJECT }}/POSTGRES_PASS
            POSTGRES_DB:${{ secrets.GOOGLE_CLOUD_PROJECT }}/POSTGRES_DB
            SQLSERVER_CONNECTION_NAME:${{ secrets.GOOGLE_CLOUD_PROJECT }}/SQLSERVER_CONNECTION_NAME
            SQLSERVER_USER:${{ secrets.GOOGLE_CLOUD_PROJECT }}/SQLSERVER_USER
            SQLSERVER_PASS:${{ secrets.GOOGLE_CLOUD_PROJECT }}/SQLSERVER_PASS
            SQLSERVER_DB:${{ secrets.GOOGLE_CLOUD_PROJECT }}/SQLSERVER_DB
      
      - name: Run tests
        env:
          MYSQL_CONNECTION_NAME: '${{ steps.secrets.outputs.MYSQL_CONNECTION_NAME }}'
          MYSQL_USER: '${{ steps.secrets.outputs.MYSQL_USER }}'
          MYSQL_PASS: '${{ steps.secrets.outputs.MYSQL_PASS }}'
          MYSQL_DB: '${{ steps.secrets.outputs.MYSQL_DB }}'
          POSTGRES_CONNECTION_NAME: '${{ steps.secrets.outputs.POSTGRES_CONNECTION_NAME }}'
          POSTGRES_IAM_CONNECTION_NAME: '${{ steps.secrets.outputs.POSTGRES_IAM_CONNECTION_NAME }}'
          POSTGRES_USER: '${{ steps.secrets.outputs.POSTGRES_USER }}'
          POSTGRES_USER_IAM: '${{ steps.secrets.outputs.POSTGRES_USER_IAM }}'
          POSTGRES_PASS: '${{ steps.secrets.outputs.POSTGRES_PASS }}'
          POSTGRES_DB: '${{ steps.secrets.outputs.POSTGRES_DB }}'
          SQLSERVER_CONNECTION_NAME: '${{ steps.secrets.outputs.SQLSERVER_CONNECTION_NAME }}'
          SQLSERVER_USER: '${{ steps.secrets.outputs.SQLSERVER_USER }}'
          SQLSERVER_PASS: '${{ steps.secrets.outputs.SQLSERVER_PASS }}'
          SQLSERVER_DB: '${{ steps.secrets.outputs.SQLSERVER_DB }}'
        # specifying bash shell ensures a failure in a piped process isn't lost by using `set -eo pipefail`
        shell: bash
        run: |
          go install github.com/jstemmer/go-junit-report/v2@latest
          go test -v -race -cover ./e2e_mysql_test.go ./e2e_postgres_test.go ./e2e_sqlserver_test.go | tee test_results.txt
          go-junit-report -in test_results.txt -set-exit-code -out integration_sponge_log.xml
        
      - name: FlakyBot (Linux)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'Linux' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot -o flakybot -s -L
          chmod +x ./flakybot
          ./flakybot --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
      - name: FlakyBot (Windows)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'Windows' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot.exe -o flakybot.exe -s -L
          ./flakybot.exe --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
      - name: FlakyBot (macOS)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'macOS' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot-darwin-amd64 -o flakybot -s -L
          chmod +x ./flakybot
          ./flakybot --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}

  unit:
    # run job on proper workflow event triggers (skip job for pull_request event from forks and only run pull_request_target for "tests: run" label)  
    if: "${{ (github.event.action != 'labeled' && github.event.pull_request.head.repo.full_name == github.event.pull_request.base.repo.full_name) || github.event.label.name == 'tests: run' }}"
    name: "unit tests"
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [macos-latest, windows-latest, ubuntu-latest]
        goarch: ["", "386"]
        go-version: [1.16, 1.17, 1.18]
        exclude:
          - os: macos-latest
            goarch: "386"
      fail-fast: false
    permissions:
      contents: 'read'
      id-token: 'write'
    env:
      GOARCH: ${{ matrix.goarch }}
    steps:
      - name: Remove PR label
        if: "${{ github.event.action == 'labeled' && github.event.label.name == 'tests: run' }}"
        uses: actions/github-script@v6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            try {
              await github.rest.issues.removeLabel({
                name: 'tests: run',
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.payload.pull_request.number
              });
            } catch (e) {
              console.log('Failed to remove label. Another job may have already removed it!');
            }
      - name: Setup Go ${{ matrix.go-version }}
        uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go-version }}
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          repository: ${{ github.event.pull_request.head.repo.full_name }}
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/[email protected]'
        with:
          workload_identity_provider: ${{ secrets.PROVIDER_NAME }}
          service_account: ${{ secrets.SERVICE_ACCOUNT }}
          access_token_lifetime: 600s
      - name: Run tests
        if: matrix.goarch == ''
        # specifying bash shell ensures a failure in a piped process isn't lost by using `set -eo pipefail`
        shell: bash
        run: |
          go install github.com/jstemmer/go-junit-report/v2@latest
          go test -v -race -cover -short ./... | tee test_results.txt
          go-junit-report -in test_results.txt -set-exit-code -out unit_sponge_log.xml
      - name: Run tests (386)
        # 386 archs don't support race detector
        if: matrix.goarch == '386'
        run: |
          go test -v -cover -short ./...      
      - name: FlakyBot (Linux)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'Linux' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot -o flakybot -s -L
          chmod +x ./flakybot
          ./flakybot --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
      - name: FlakyBot (Windows)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'Windows' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot.exe -o flakybot.exe -s -L
          ./flakybot.exe --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}
      - name: FlakyBot (macOS)
        # only run flakybot on periodic (schedule) and continuous (push) events
        if: ${{ (github.event_name == 'schedule' || github.event_name == 'push') && runner.os == 'macOS' && always() }}
        run: |
          curl https://github.com/googleapis/repo-automation-bots/releases/download/flakybot-1.1.0/flakybot-darwin-amd64 -o flakybot -s -L
          chmod +x ./flakybot
          ./flakybot --repo ${{github.repository}} --commit_hash ${{github.sha}} --build_url https://github.com/${{github.repository}}/actions/runs/${{github.run_id}}

Log output

https://github.com/GoogleCloudPlatform/cloud-sql-go-connector/runs/8254487502?check_suite_focus=true

Additional information

No response

Sep 08 '22 17:09 jackwotherspoon

Hi there @jackwotherspoon :wave:!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

Sep 08 '22 17:09 github-actions[bot]

Hi @jackwotherspoon

Can you please provide the GitHub Actions debug logs (more details in the troubleshooting guide link above). I realize it's not happening all the time, but having the debug logs will be very helpful for us to dig in further.

Sep 08 '22 19:09 sethvargo

Happened for us also for the first time using google-github-actions/auth@v1 and service account federation.

Unfortunately we also don't have debug logs, it failed suspiciously fast (so network timeout is unlikely).

2023-04-13T11:48:39.7298240Z ##[group]Run google-github-actions/auth@v1
2023-04-13T11:48:39.7298513Z with:
<configuration removed from public post>
2023-04-13T11:48:39.7302701Z ##[endgroup]
2023-04-13T11:48:39.8373430Z ##[error]google-github-actions/auth failed with: retry function failed after 1 attempt: error message: Cannot read properties of undefined (reading 'message')

Google cloud logs have nothing about this exact timestamps, but we have successful token exchanges from the same workflow (it is matrix with multiple concurrent jobs at same time) second earlier and second later.

Apr 13 '23 12:04 ervin-pactum

auth auth copied to clipboard

Error: google-github-actions/auth failed with: retry function failed after 1 attempt: error message: Cannot read properties of undefined (reading 'message')

TL;DR

Expected behavior

Observed behavior

Action YAML

Log output

Additional information

auth
auth copied to clipboard