firebase-tools Cannot deploy V2 Cloud Functions with more than 8 GiB of memory

[REQUIRED] Environment info

firebase-tools: ^11.4.1

Platform: Ubuntu

[REQUIRED] Test case

You can reproduce the error by defining a V2 Cloud Function as below:

import { onCall } from 'firebase-functions/v2/https'

export const dummycf = onCall({
  timeoutSeconds: 3600,
  memory: "16GiB",
  cpu: 2
}, (request) => {
  console.log(`This V2 CF w/ 16GiB of memory and 2 vCPUs should work fine!`)
})

[REQUIRED] Steps to reproduce

You just need to deploy the Cloud Function by running:

firebase deploy --only functions:test

[REQUIRED] Expected behavior

The Cloud Function must be correctly deployed with 4 vCPUs and 16 GiB of memory. The V2 public preview should support the memory and cpu values as reported in the documentation.

[REQUIRED] Actual behavior

Running the command above results in the following error:

> firebase deploy --only functions:dummycf

=== Deploying to 'mpc-phase2-suite-test'...

i  deploying functions
Running command: npm --prefix "$RESOURCE_DIR" run build

> [email protected] build
> tsc

✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: Loaded environment variables from .env.

Error: available memory must be a supported value, if present

Aug 01 '22 17:08 0xjei

Hey @0xjei - thanks for reporting and apologies for the issue you are seeing.

I tried giving 16GiB function a spin and it gave my setup no problem:

exports.sixteen = functionsv2.https.onCall({ memory: "16GiB", cpu: 2 }, () => { return { foo: "bar"}; });

Can you check to see if you are using the latest version of the firebase-functions SDK? In my setup, I was using 3.22.0.

Aug 02 '22 23:08 taeold

Hey @taeold, thanks for your kind and quick response.

Uhm, yeah. I'm using the same version of firebase-functions but I'm still getting the error when trying to run firebase:deploy or firebase:deploy-functions scripts. Also, I have checked the deploy region.

You can see the list of the scripts and packages in my package.json here:

  "scripts": {
    "build": "tsc",
    "firebase:login": "firebase login",
    "firebase:logout": "firebase logout",
    "firebase:init": "firebase init",
    "firebase:deploy": "yarn firestore:get-indexes && firebase deploy",
    "firebase:deploy-functions": "firebase deploy --only functions",
  },
 "devDependencies": {
    "@firebase/rules-unit-testing": "^2.0.3",
    "@types/uuid": "^8.3.4",
    "firebase-functions-test": "^2.2.0",
    "firebase-tools": "^11.4.1",
    "typescript": "^4.7.3"
  },
  "dependencies": {
    "blakejs": "^1.2.1",
    "dotenv": "^16.0.1",
    "firebase-admin": "^11.0.1",
    "firebase-functions": "^3.22.0",
    "snarkjs": "^0.4.22",
    "timer-node": "^5.0.5",
    "uuid": "^8.3.2",
    "winston": "^3.7.2"
  }

Just to let you know, I have correctly deployed the function with 16GiB from the Google Cloud Function Console manually, but for what I'm working on I still need the automation from your library.

UPDATE: I have tried with a new project from scratch and it seems to work properly. Now, I'm trying to deploy using --debug flag to see what's going on. Keep you updated.

PARTIALLY SOLVED By using the firebase init command from the CLI to create a Cloud Functions only project, returns a simple helloWorld function. Switching to v2, I noticed that the import is different from what I was doing initially:

// From the `firebase init`.
import * as functions from "firebase-functions/v2";
// From my issue.
import { onCall } from 'firebase-functions/v2/https'

So, I tried this import on my issue above and now it works. Also, I have switched from cpu: 2 to 4 because 2 was not suitable for 16 GiB of memory.

import * as functions from "firebase-functions/v2";

export const dummycf = functions.https.onCall({
  timeoutSeconds: 3600,
  memory: "16GiB",
  cpu: 4
}, (request) => {
  console.log(`This V2 CF w/ 16GiB of memory and 2 vCPUs should work fine!`)
})

In any case, shouldn't it work both ways? However, I'm noticing that the deploy with V2 seems inconsistent: sometimes the deployment works fine, sometimes requires to deploy the function with V1 settings (e.g., 8GiB max) or crashes with CPU/Memory errors (which aren't correct since retrying with the same settings works fine).

Hope it helps and thanks again @taeold for the support! :+1:

Aug 03 '22 08:08 0xjei

@0xjei Yes it should be working in both cases. Thank you for your detailed feedback - I'll go back and see what I can uncover.

Aug 03 '22 14:08 taeold

I played around with several settings, and at least locally on my machine:

import { onCall } from "firebase-functions/v2/https";
export const shake = onCall({ memory: "16GiB"}, () => "hello");

import * as fv2 from "firebase-functions/v2";
export const wild = fv2.https.onCall({ memory: "16GiB" }, () => "hello");

Both successfully deployed a 16 GiB function:

That said, I think there are several inconsistencies that can be introduced in the version of the Firebase CLI and Functions SDK isn't compatible. For example, order versions of the Functions SDK or CLI won't support 16GiB memory option and might lead to inconsistent deployment behavior.

There are rough edges here - I think we will have to, at some point, make a breaking change in the CLI to emit errors like "Please use Functions SDK version > X.X.X" instead of leaving things in consistent state as you've helped us discover. Please stay tuned.

Until then - do you mind double checking your usage with import { onCall } from "firebase-functions/v2/https"; style and letting me know how it goes. It would be great if you can share a little more about your setup (e.g. if you are using TypeScript, sharing some parts of tsconfig.json would be super helpful for debugging since I can't reproduce the error yet).

Aug 03 '22 17:08 taeold

Hi @taeold! I appreciate the effort and clarity of your response. Anyway, I may have found the problem here and I think this is closely related to this PR #4823.

As you know, I have been trying different combinations of memory + cpu for CF V2 these days without success. When I found this PR, I noticed that when I was importing the onCall() method I was configuring the function with just 16 GiB as memory and no CPU option set, leading to the invalid CPU setting error that you've found in the PR.

So, just adding a cpu: 4 option the deploy with both import styles works fine! :tada:

Anyway, something strange happens when I deploy the functions for the first time: everything seems correct because I can see the functions in the dashboard without no errors and when I call them everything works fine and I can see logs and results. However, the deploy script got me back this error:

npx firebase deploy --only functions:helloworld

=== Deploying to 'mpc-phase2-suite-test'...

i  deploying functions
Running command: yarn --prefix "$RESOURCE_DIR" build
yarn run v1.22.18
$ tsc
Done in 2.12s.
✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: Loaded environment variables from .env.
i  functions: preparing . directory for uploading...
i  functions: packaged /code/mpc-phase2-suite/firebase (227.87 KB) for uploading
i  functions: packaged /code/mpc-phase2-suite/firebase (228.1 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: . folder uploaded successfully
i  functions: creating Node.js 16 function helloworld(us-central1)...
Failed to update Run service projects/mpc-phase2-suite-test/locations/us-central1/services/helloworld

Functions deploy had errors with the following functions:
        helloworld(us-central1)
i  functions: cleaning up build files...

Error: There was an error deploying functions

And you can see more clearly the error from the debug logs:

// Everything seems okay before this line...
[error] Failed to update Run service projects/mpc-phase2-suite-test/locations/us-central1/services/helloworld
[debug] [2022-08-04T10:19:48.823Z] Total Function Deployment time: 198044
[debug] [2022-08-04T10:19:48.823Z] 1 Functions Deployed
[debug] [2022-08-04T10:19:48.823Z] 1 Functions Errored
[debug] [2022-08-04T10:19:48.823Z] 0 Function Deployments Aborted
[debug] [2022-08-04T10:19:48.823Z] Average Function Deployment time: 198043
[info] 
[info] Functions deploy had errors with the following functions:
	helloworld(us-central1)
[info] i  functions: cleaning up build files... 
[debug] [2022-08-04T10:19:48.835Z] >>> [apiv2][query] DELETE https://artifactregistry.googleapis.com/v1beta2/projects/mpc-phase2-suite-test/locations/us-central1/repositories/gcf-artifacts/packages/helloworld [none]
[debug] [2022-08-04T10:19:49.331Z] Could not find image for function projects/mpc-phase2-suite-test/locations/us-central1/functions/helloworld
[debug] [2022-08-04T10:19:49.752Z] <<< [apiv2][status] DELETE https://artifactregistry.googleapis.com/v1beta2/projects/mpc-phase2-suite-test/locations/us-central1/repositories/gcf-artifacts/packages/helloworld 200
[debug] [2022-08-04T10:19:50.457Z] Functions deploy failed.
[debug] [2022-08-04T10:19:50.457Z] {
  "endpoint": {
    "id": "helloworld",
    "project": "mpc-phase2-suite-test",
    "region": "us-central1",
    "entryPoint": "helloworld",
    "platform": "gcfv2",
    "runtime": "nodejs16",
    "callableTrigger": {},
    "labels": {
      "deployment-tool": "cli-firebase"
    },
    "availableMemoryMb": 32768,
    "environmentVariables": {
      "FIRESTORE_DATABASE_URL": "....firebaseio.com", // not publicly available
      "PROJECT_ID": "...", not publicly available
      "STORAGE_BUCKET": "....appspot.com", // not publicly available
      "SCHEDULER_START_CEREMONY": "20",
      "SCHEDULER_STOP_CEREMONY": "20",
      "FIRST_ZKEY_INDEX": "00000",
      "TIMEOUT_TOLERANCE_RATE": "20",
      "RETRY_WAITING_TIME_IN_DAYS": "1",
      "CUSTOM_CLAIMS_COORDINATOR_EMAIL_ADDRESS_OR_DOMAIN": "[email protected]", // not publicly available
      "FIREBASE_CONFIG": "{\"projectId\":\"...\",\"storageBucket\":\"....appspot.com\",\"locationId\":\"...\"}", not publicly available
      "GCLOUD_PROJECT": "..." not publicly available
    },
    "codebase": "default",
    "cpu": 4,
    "uri": "https://helloworld-43d5rlynya-uc.a.run.app",
    "concurrency": 80
  },
  "op": "set concurrency",
  "original": {
    "name": "FirebaseError",
    "children": [],
    "exit": 1,
    "message": "Failed to update Run service projects/mpc-phase2-suite-test/locations/us-central1/services/helloworld",
    "original": {
      "name": "FirebaseError",
      "children": [],
      "context": {
        "body": {
          "error": {
            "code": 400,
            "message": "spec.template.spec.containers[0].resources.limits.memory: Invalid value specified for memory. For 4.0 CPU, memory must be between 2Gi and 16Gi inclusive.\nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/configuring/memory-limits",
            "status": "INVALID_ARGUMENT",
            "details": [
              {
                "@type": "type.googleapis.com/google.rpc.BadRequest",
                "fieldViolations": [
                  {
                    "field": "spec.template.spec.containers[0].resources.limits.memory",
                    "description": "Invalid value specified for memory. For 4.0 CPU, memory must be between 2Gi and 16Gi inclusive.\nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/configuring/memory-limits"
                  }
                ]
              }
            ]
          }
        },
        "response": {
          "statusCode": 400
        }
      },
      "exit": 1,
      "message": "HTTP Error: 400, spec.template.spec.containers[0].resources.limits.memory: Invalid value specified for memory. For 4.0 CPU, memory must be between 2Gi and 16Gi inclusive.\nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/configuring/memory-limits",
      "status": 400
    },
    "status": 500
  }
}
[debug] [2022-08-04T10:19:50.458Z] Error: Failed to set concurrency function helloworld in region us-central1
    at /code/mpc-phase2-suite/node_modules/firebase-tools/lib/deploy/functions/release/fabricator.js:41:11
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Fabricator.setRunTraits (/code/mpc-phase2-suite/node_modules/firebase-tools/lib/deploy/functions/release/fabricator.js:388:9)
    at async Fabricator.createV2Function (/code/mpc-phase2-suite/node_modules/firebase-tools/lib/deploy/functions/release/fabricator.js:283:13)
    at async Fabricator.createEndpoint (/code/mpc-phase2-suite/node_modules/firebase-tools/lib/deploy/functions/release/fabricator.js:121:13)
    at async handle (/code/mpc-phase2-suite/node_modules/firebase-tools/lib/deploy/functions/release/fabricator.js:77:17)
[error] 
[error] Error: There was an error deploying functions

On the other hand, when I "update" them (i.e., function is deployed and I upgrade the memory or CPU settings), I get no error as you can noticed from the "update" log:

npx firebase deploy --only functions:helloworld

=== Deploying to 'mpc-phase2-suite-test'...

i  deploying functions
Running command: yarn --prefix "$RESOURCE_DIR" build
yarn run v1.22.18
$ tsc
Done in 2.14s.
✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: preparing codebase default for deployment
i  functions: Loaded environment variables from .env.
i  functions: preparing . directory for uploading...
i  functions: packaged /code/mpc-phase2-suite/firebase (227.87 KB) for uploading
i  functions: packaged /code/mpc-phase2-suite/firebase (228.1 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: . folder uploaded successfully
i  functions: updating Node.js 16 function helloworld(us-central1)...
✔  functions[helloworld(us-central1)] Successful update operation.
i  functions: cleaning up build files...

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/mpc-phase2-suite-test/overview

Also, I agree w/ you in making the user aware of the SDK upgrades "Please use Functions SDK version > X.X.X". I can add something too here: updating the firebase-admin package to the latest version (v11.0.1 rn) helped me to avoid inconsistencies when passing memory and CPU configs (because I'm using the library to initialize the app).

I hope it can help you to close the loop and thanks again for the support and patience :1st_place_medal:

Aug 04 '22 10:08 0xjei

@0xjei Thanks again for detailed bug report and your patience being on the bleeding edge. Will get back at the create issue soon,

Aug 04 '22 12:08 taeold

@0xjei Sorry I didn't get back earlier.

I reproduced a similar error to yours in my local setup:

import { onRequest } from "firebase-functions/v2/https";
export const v = onRequest({ memory: "16GiB", cpu: 2  }, (req, res) => { res.sendStatus(200); });

16GiB + 2 CPU. I get following error:

Invalid value specified for memory. For 2.0 CPU, memory must be between 128Mi and 8Gi inclusive.

For more troubleshooting guidance, see https://cloud.google.com/run/docs/configuring/memory-limits

So it turns out 2 CPU + 16 GiB is not an allowed configuration for Cloud Run. This is a surprised - I don't see anything about *maximum memory per CPU shape.

Looking again at your error message:

Invalid value specified for memory. For 4.0 CPU, memory must be between 2Gi and 16Gi inclusive.

Looks like your function is configured w/ 32 GiB which isn't allowed to have 4.0 CPU - it must have 8.0 CPU or more.

I think we still have couple of bugs on incorrect mapping of memory -> CPU that I'm going to try to fix here. Thanks for helping us find the bug.

(PS: I'm not sure why the subsequent update is working for you 🤔. I'm going to focus on fixing the bug for now)

Aug 09 '22 04:08 taeold

Hi, I'm also had a similar issue - Running with firebase-functions version 3.22.0, and I am running this:

export const func = functionsV2.https.onCall(
  timeoutSeconds: 1800,
  memory: '16GiB',
  cpu: 4,
}
...

And I was getting: Error: available memory must be a supported value, if present

The problem was that I used firebase-tools in version 11.3.0. Upgrading it to 11.6.0 solved the issue.

Aug 15 '22 09:08 Omerr

Hey, @taeold! Thank you for your explanations and responsiveness in solving the issue. Glad to have helped you guys finding the bug.

BTW, after your fix and upgrading to 11.7.0 version everything seems working fine when deploying combinations of 16GiB/4CPUs and 32GiB/8CPUs :tada:

Aug 16 '22 11:08 0xjei

Great to hear! Thanks for being proactive and helping up squash many bugs! 🐛

Aug 16 '22 17:08 taeold

firebase-tools firebase-tools copied to clipboard

Cannot deploy V2 Cloud Functions with more than 8 GiB of memory

[REQUIRED] Environment info

[REQUIRED] Test case

[REQUIRED] Steps to reproduce

[REQUIRED] Expected behavior

[REQUIRED] Actual behavior

firebase-tools
firebase-tools copied to clipboard