firebase-tools

firebase-tools copied to clipboard

[REQUIRED] Environment info

firebase-tools: 10.1.4

We are not able to upgrade to the most recent version, as it breaks the emulator suite. I saw that this issue was supposed to be fixed in the most recent release, however the emulators still fail to start up with the most recent version. Our functions config values are still not being loaded in properly when we start the emulators, so we are using 10.1.4 until further notice.

Platform: Docker image cimg/node:16.14.0 via CircleCI

[REQUIRED] Test case

Here is error output from our CircleCI build:

i  functions: updating Node.js 16 function functionName1(us-central1)...
i  functions: updating Node.js 16 function functionName2(us-central1)...
i  functions: updating Node.js 16 function functionName3(us-central1)...

Functions deploy had errors with the following functions:
	functionName1(us-central1)
	functionName2(us-central1)
	functionName3(us-central1)
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName1 in region us-central1
- Error Failed to update function functionName2 in region us-central1
- Error Failed to update function functionName3 in region us-central1

Exited with code exit status 123
CircleCI received exit code 123

When looking at the function logs, I have seen a variety of vague errors that I cannot find the cause of. Here is one of these errors:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 14,
      "message": "The service has encountered an error during container import. Please try again later"
    },
    "authenticationInfo": {
      "principalEmail": "REDACTED"
    },
    "serviceName": "cloudfunctions.googleapis.com",
    "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
    "resourceName": "REDACTED"
  },
  ...
  "severity": "ERROR",
  ...
}

And here is another:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 9,
      "message": "Invalid source token"
    },
    "authenticationInfo": {
      "principalEmail": "REDACTED"
    },
    "serviceName": "cloudfunctions.googleapis.com",
    "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
    "resourceName": "REDACTED"
  },
  "severity": "ERROR",
  ...
}

These logs are preceded by about a dozen protoPayload.methodName: google.longrunning.Operations.GetOperation calls in the function logs. These errors appear consistently and reproducibly.

[REQUIRED] Steps to reproduce

Run a function deployment from CircleCI using Firebase version 9.23.3 or later (this is the first version we encountered issues with, and continued encountering issues after upgrading) and Node version 16.14.0.

Deploying from the command line works fine.

[REQUIRED] Expected behavior

The CircleCI deployment succeeds and all functions update successfully as they did before the firebase-tools upgrade.

[REQUIRED] Actual behavior

The deployment fails with the errors above.

@trex-quo Thanks for reporting the issue.

Out of curiosity - is it possible that you may be running multiple deploys in parallel? I think another user reported the same error message ("Invalid source token"), and the reason was because they had multiple deploys running in parallel, resulting in the post-deploy container image cleanup to delete images that are still needed in another, on-going deploy.

@trex-quo Thanks for reporting the issue.

Out of curiosity - is it possible that you may be running multiple deploys in parallel? I think another user reported the same error message ("Invalid source token"), and the reason was because they had multiple deploys running in parallel, resulting in the post-deploy container image cleanup to delete images that are still needed in another, on-going deploy.

Thanks for the reply! We are running multiple deploys in parallel yes. If I remember correctly, an older version of firebase tools had a maximum directory size upload limit, which I found frequently errored if I deployed more than ~8 functions at a time. We have parallel builds deploying about 8 functions at once.

I have noticed that after updating firebase tools that we can deploy many more functions at once, and are only bounded by the quota of 80 functions per 100 seconds. We do have more than 80 cloud functions, so do you recommend that we change our CI system to do sequential deploys of 80 functions at a time instead of doing concurrent deployments? Also, do you consider this to be a bug with Firebase function deployments, or can we expect this behavior to remain as a part of the system moving forward?

If I can provide any additional information here, such as versions of firebase-tools that we switched from, or our CircleCI build command, please let me know.

@trex-quo Thanks for a detailed reply.

The Firebase CLI is tuned to deploy in batches of 40 functions at any given time, so I suspect issuing a single firebase deploy command to deploy all your 80+ functions should be much better than your expectation. For users like you, we may consider opening up the knobs so you can tune the concurrency that fits your need, but we emply some tricks (e.g. re-using container images built for one function for another function deploy) and overall I think we can satisfy 90% of the use case w/ current set of parameters.

I recommend that you give a single deploy command a spin. I'd be curious if your deploy time seriously degrade after this change - please let us know what happens. I'd be more than excited to learn and figure out a way to improve experiences that involves large # of function deploys.

@taeold

I did try a single firebase deploy --only functions command a couple weeks back, but ran into a separate error which prevented the full deployment from going through. It is the same issue as on this thread:

https://github.com/firebase/firebase-tools/issues/3919#issuecomment-979788970

One comment on the thread by @abeisgoat says:

Hey we're aware of issues when deploying over 100 functions, it's best to deploy the functions you changed specifically instead of the whole lot when you're at this scale.

which seems to suggest that it is known that deploying all functions in a single command won't work.

That being said, I did just try another full deployment now with ~110 functions and it appeared to go through successfully. I have deleted some unused functions since the last time I tried this (it was probably ~140 functions at the time). Since our pipeline is failing now anyway, I will try replacing our CI system with a single firebase deploy --only functions command and monitor the rate at which it fails and succeeds over the coming weeks.

Here is an instance where a "Quota exceeded" error failed the deployment. Since merging the change into our CI that deploys all functions in a single command, the success rate seems about 50%. Here is the full (redacted) output of the deployment.

Now using project project-name

=== Deploying to 'project-name'...

i  deploying functions
Running command: predeploy_script.js
Currently in project-name. Setting flags and deploying...

> lint
> eslint .
> build

✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (3.12 MB) for uploading
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: functions folder uploaded successfully

The following functions are found in your project but do not exist in your local source code:
	functionName(us-central1)
	(~10 more functions listed)

If you are renaming a function or changing its region, it is recommended that you create the new function first before deleting the old one to prevent event loss. For more info, visit https://firebase.google.com/docs/functions/manage-functions#modify

? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) ? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "n? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "nn? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "nn"? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. No

i  functions: updating Node.js 16 function functionName(us-central1)...
(repeated for ~70 more functions)

Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
(repeated for ~25 more functions)

⚠  functions: got "Quota Exceeded" error while trying to update projects/project-name/locations/us-central1/functions/functionName. Waiting to retry...
(repeated for ~50 more functions

✔  functions[functionName2(us-central1)] Successful update operation.
(repeated for ~20 more functions)

Functions deploy had errors with the following functions:
	functionName3(us-central1)
	(repeated for ~30 more functions)

Function URL (functionName4(us-central1)): *functionURL*
(repeated for ~50 more functions)

i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName in region us-central1
(repeated for ~50 more functions)

Exited with code exit status 2
CircleCI received exit code 2

@trex-quo Thanks for the feedback. 50% success rate is terrible, and we should strive to do better. We'll work on tuning our parameters to deal w/ quota issues better.

In the immediate terms, it sounds like breaking up deploys as you've done before but NOT running them in parallel might be most reliable thing to do.

We recently added a group to our deployments and we receive this error 100% of the time when deploying all of the functions. 36 functions in total, 9 of which are in a group.

I haven't been able to find a workaround yet. It doesn't look like I can independently deploy the default functions.

This happens to me consistently with a function registered through functions.tasks.taskQueue during deployment of all functions (firebase deploy --only functions). All other functions get deployed except that one. Looks like it doesn't happen during the first fresh deployment when the problematic function doesn't exist yet. After it has been deployed, subsequent attempts to deploy that function fail. My firebase-tools version is 11.1.0

A workaround that works for me: delete the function before each deployment.

@tojtoj That's pretty much what was happening to me. Upgrading to 11.4.0 fixed the "Invalid source token" error. ~~It also brought other error (https://github.com/firebase/firebase-tools/issues/4730), but I fixed it with https://npm.im/patch-package.~~

EDIT: https://github.com/firebase/firebase-tools/issues/4730 is fixed in 11.4.2.

Still the issue even in 11.6.1 (node 14.21):

Failed 8 deployments in a row. Tired of retrying.

I have 29 functions and last 4 always fail now.

It's a nightmare, wasted so much of our team's time because of this. Fails even on 8 functions

Any updates on this?

@inlined @taeold

@n-sviridenko Sorry we missed your message here.

Do you mind sharing firebase-debug.log for a failed deploy? One of the fix we introduced earlier last month was aimed at fixing the issue when the deployment time exceeds >30min. When deployment time gets that long, the you'd see an error message like "Invalid source token".

I'm going to try to reach out to the GCF team internally to see if they have any recent changes that may impact the GCF deployment that relies on source token.

Apologies for the troubles you are facing.

Hey @trex-quo. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

Since there haven't been any recent updates here, I am going to close this issue.

@trex-quo if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

Hi @taeold , it fails even after ~2-3 minutes. At the moment when I wrote the first message here, we split function deployments into chunks of 4 and they were still failing. Nothing has changed since then. The 30+ min fix is irrelevant.

@trex-quo @google-oss-bot we need to reopen this issue as it's not fixed yet

And there was no firebase-debug.log generated after this error (neither in the root folder where firebase.json is nor inside functions folder).

Since there haven't been any recent updates here, I am going to close this issue.

@trex-quo if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

@taeold the bot closed it again, seems like it's broken

@taeold I checked the logs and sometimes we have this error: The service has encountered an error during container import. Please try again later

Here's a video overview: https://www.loom.com/share/9afb2facb5e3461ebef74e7e802a2761

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 14,
      "message": "The service has encountered an error during container import. Please try again later"
    },
    "authenticationInfo": {},
    "serviceName": "cloudfunctions.googleapis.com",
    "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
    "resourceName": "projects/voypost-matching-prod/locations/europe-west3/functions/createFromJobPubSub"
  },
  "insertId": "-wq3kwnb2c",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "voypost-matching-prod",
      "function_name": "createFromJobPubSub",
      "region": "europe-west3"
    }
  },
  "timestamp": "2023-01-07T16:39:01.688761Z",
  "severity": "ERROR",
  "logName": "projects/voypost-matching-prod/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "operations/dm95cG9zdC1tYXRjaGluZy1wcm9kL2V1cm9wZS13ZXN0My9jcmVhdGVGcm9tSm9iUHViU3ViL0pOaGkxYW9zMWxj",
    "producer": "cloudfunctions.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2023-01-07T16:39:02.021123649Z"
}

As per this doc, it may be due to non-utf8 characters, but there are none (checked using https://stackoverflow.com/a/41741313 grep -axv '.*' ./lib/**/*.js)

It failed 3 times in a row and continues failing. And every time it fails on the same functions.

@n-sviridenko Sorry for the troubles. Few questions about your setup:

1. Are you running multiple deployments in parallel (e.g. multiple firebase deploy commands running at the same time?)

The CLI is not "multi-deploy" safe since after every CLI deployment, it will go ahead and try to delete container images associated with the deployment to reduce the impact of container storage cost on your GCP account.

2. If possible, please contact GCP Support. They should route you to the engineering team that would have access to internal logs to help debug the issue further. Unfortunately, I don't have access to customer logs in Google Cloud Functions/Cloud Run and can't get any more information beyond what you are seeing in the logs.

1. No, there is always only one deployment ongoing

2. Seems like this option is paid so we don't have access to it. Can you create a case for us?

cc @taeold

@taeold also, we have the "Invalid source token" error again. Deployment took just 4m23s, so it's not about the 30m limit. There is no firebase-debug.log generated. Firebase tools are 11.16.1.

Hi @taeold , just wanted to follow-up on this