firebase-functions icon indicating copy to clipboard operation
firebase-functions copied to clipboard

Firebase Functions v2: Cloud Scheduler receives PERMISSION_DENIED even with correct Cloud Run URLs and IAM permissions

Open ebeloded opened this issue 2 months ago • 4 comments

[REQUIRED] Environment info

firebase-functions: 6.4.0
firebase-tools: 14.16.0
Platform: macOS (Darwin 25.0.0)

[REQUIRED] Test case

import { onSchedule } from 'firebase-functions/v2/scheduler'

// Deploy this function
export const testScheduler = onSchedule(
  {
    schedule: 'every 2 minutes',
    timeZone: 'America/Los_Angeles',
  },
  async () => {
    console.log('Test scheduler executed at', new Date().toISOString())
  }
)

[REQUIRED] Steps to reproduce

  1. Delete any existing scheduled functions: firebase functions:delete FUNCTION_NAME --force
  2. Deploy a fresh v2 scheduled function: firebase deploy --only functions
  3. Check Cloud Scheduler console - job is created with correct Cloud Run URL
  4. Wait for scheduled execution or manually trigger
  5. Cloud Scheduler reports PERMISSION_DENIED error
  6. However, manually invoking the Cloud Run URL with proper auth token works:
TOKEN=$(gcloud auth print-identity-token --impersonate-service-account=SERVICE_ACCOUNT --audiences=CLOUD_RUN_URL)
curl -X POST CLOUD_RUN_URL -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{}'
# Returns 200 OK

[REQUIRED] Expected behavior

Cloud Scheduler should successfully invoke the v2 scheduled functions without permission errors.

[REQUIRED] Actual behavior

  • Cloud Scheduler jobs are created with correct Cloud Run URLs (.run.app format)
  • Cloud Run services have correct IAM bindings (roles/run.invoker for the service account)
  • Scheduler jobs have correct OIDC token configuration
  • BUT all scheduled invocations fail with PERMISSION_DENIED
  • Direct invocation with the same service account credentials works fine

Details and Analysis

After investigation, the issue appears to be with how Firebase sets up the authentication chain between Cloud Scheduler and Cloud Run for v2 functions:

  1. Cloud Scheduler Configuration (looks correct):
{
  "httpTarget": {
    "uri": "https://FUNCTION_NAME-HASH-uc.a.run.app/",
    "httpMethod": "POST",
    "headers": {
      "User-Agent": "Google-Cloud-Scheduler"
    },
    "oidcToken": {
      "serviceAccountEmail": "[email protected]",
      "audience": "https://FUNCTION_NAME-HASH-uc.a.run.app"
    }
  }
}
  1. Cloud Run IAM Policy (looks correct):
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/run.invoker
  1. Yet the error persists:
{
  "@type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
  "jobName": "projects/PROJECT/locations/REGION/jobs/firebase-schedule-FUNCTION",
  "status": "PERMISSION_DENIED",
  "targetType": "HTTP",
  "url": "https://FUNCTION_NAME-HASH-uc.a.run.app/"
}

Affected Functions State

  • Functions are in ACTIVE state
  • Cloud Run services exist and are healthy
  • Manual invocation with identity tokens works
  • Only Cloud Scheduler invocation fails

Workaround Attempts That Don't Work

  1. Deleting and redeploying functions
  2. Using object syntax for onSchedule() (fixes URL issue from #1734 but not auth)
  3. Waiting for propagation (issue persists after hours)

Potential Root Causes

  1. The App Engine default service account may need additional permissions to create identity tokens for itself
  2. There may be a missing step in the Firebase deployment process for setting up the OAuth flow
  3. The OIDC audience field might need special formatting for v2 functions

Impact

This makes Firebase Functions v2 scheduled functions completely unusable in production, as they cannot be invoked by Cloud Scheduler despite appearing to be correctly configured.

Related Issues

  • #1734 (URL format issue - fixed by using object syntax but auth still fails)
  • #1425 (General PERMISSION_DENIED errors)
  • #1447 (Cloud Function V2 URL issues)

This is a critical issue as it prevents migration from v1 to v2 for any scheduled functions.

ebeloded avatar Sep 25 '25 19:09 ebeloded

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

google-oss-bot avatar Sep 25 '25 19:09 google-oss-bot

Hi! Thanks for raising this. It's been received and we will provide updates when available.

cabljac avatar Sep 29 '25 16:09 cabljac

@cabljac Have there been any updates on this in the last quarter? I'm stuck with none of my functions working after migrating to functions v2 and no workaround in sight

swantzter avatar Dec 10 '25 08:12 swantzter

I tried digging into this a little, and like @ebeloded says the cloud scheduler job is created pointing at the URL https://<function-name>-<nonce>-<short-region>.a.run.app/

However looking at the cloudrun functions security settings it has a custom audiences value of https://<region>-<project>.cloudfunctions.net/<function-name>

I tried modifying a scheduler to set the audience value to https://<region>-<project>.cloudfunctions.net/<function-name>, I also tried setting the values for timezone and max doublings which the cloud scheduler console UI complained were missing, and lastly I tried setting the invocation URL to https://<region>-<project>.cloudfunctions.net/<function-name> as well, with no success

swantzter avatar Dec 11 '25 08:12 swantzter