Cloud tasks stopped working as of yesterday: "failed to decode protobuf and create a before snapshot"

[nederev]

I'm using google tasks with firebase.

1. Emulators affected
2. Production affected

• Firebase CLI version 13.1.0 and lower (tried 12.x)
• NodeJS version: 18 or 20 both fail
• @google-cloud/tasks version: 5.0.0 or 4.0.1 both fail

As soon as I include those lines of code in my nodejs project i get fatal errors:

code to include: import { CloudTasksClient } from '@google-cloud/tasks'; const tasksClient = new CloudTasksClient();

that's enough for everything to stop working and fail with errors:

> {"severity":"ERROR","message":"Error: Failed to decode protobuf and create a before snapshot.\n at entryFromArgs (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/logger/index.js:130:19)\n at Object.error (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/logger/index.js:116:11)\n at createBeforeSnapshotFromProtobuf (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/common/providers/firestore.js:77:16)\n at createBeforeSnapshot (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:134:65)\n at makeChangedFirestoreEvent (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:172:37)\n at func (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:238:32)\n at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:524:16\n at runFunction (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:506:15)\n at runCloudEvent (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:523:11)\n at processBackground (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:490:16)"} ⚠ functions: TypeError: Cannot read properties of undefined (reading 'cloud') at Function.decode (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/protos/compiledFirestore.js:1529:130) at createBeforeSnapshotFromProtobuf (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/common/providers/firestore.js:73:52) at createBeforeSnapshot (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:134:65) at makeChangedFirestoreEvent (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:172:37) at func (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:238:32) at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:524:16 at runFunction (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:506:15) at runCloudEvent (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:523:11) at processBackground (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:490:16) at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:690:27 ⚠ Your function was killed because it raised an unhandled error.

[nederev]

Found working version of @google-cloud/tasks: 4.0.0

[sofisl]

Is this being compiled from TS? If so, could you provide the TS configuration as well?

In 5.0.0, we published Cloud Tasks to both ESM and CJS, so it probably has something to do with that. Testing locally, it works fine if the module is ESM and if it's being compiled from TS, but of course that can depend on your settings.

[nederev]

here's tsconfig:

{
  "include": [ "src" ],
  "exclude": [
    "src/just/example",
  ],
  "compilerOptions": {
    "noImplicitAny": true,
    "esModuleInterop": true,
    "outDir": "functions",
    "module": "commonjs",
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "sourceMap": true,
    "strict": true,
    "target": "es2017",
    "moduleResolution": "node",
    "typeRoots": ["node_modules/@types"]  // <-- here
  },
  "compileOnSave": true,
}

[LeandroTorresSicilia]

This is also happening to me since January 29. In my case, I use Firebase cloud functions and one function uses cloud task, now all the other functions stopped working because of this issue.

[LeandroTorresSicilia]

I tried versions 5.0.0, 4.0.1, and 4.0.0 and get the same error

[nederev]

now all the other functions stopped working because of this issue.

yep, all functions will stop working, simple importing import { CloudTasksClient } from '@google-cloud/tasks'; will break everything

I tried versions 5.0.0, 4.0.1, and 4.0.0 and get the same error

this is very strange, try to remove node_modules, remove functions directory etc, 4.0.0 works for me right now,

here's my package.json maybe it will help

    @google-cloud/tasks": "4.0.0",
    "firebase-admin": "^11.11.1",
    "firebase-functions": "^4.6.0",
    
  "engines": {
    "node": "20"
  },

[LeandroTorresSicilia]

@olegdater It worked, thanks!!! I had ^4.0.0 instead of 4.0.0. Now it works.

[sofisl]

Thank you both for submitting this issue! I've attempted to reproduce but I can't seem to do it, which leads me to believe there's something with Firebase going on....

I created a simple reproduction with @olegdater's tsconfig, and this works for me: https://github.com/sofisl/test-repo-tasks/tree/main. Can you confirm if this works for you, by cloning the repo, then running:

npm i && npx tsc && node functions/index.js projects/YOUR-PROJECT/locations/YOUR-LOCATION

If it works, I think we have a Firebase smoking gun!

[nederev]

Runs without errors:

?1 ~/Projects/test-repo-tasks npm i && npx tsc && node functions/index.js projects/humanornot-dev/locations/us-central1

> [email protected] prepare
> npm run compile


> [email protected] compile
> tsc


up to date, audited 329 packages in 2s

63 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
?1 ~/Projects/test-repo-tasks         

[sofisl]

Ok, we're on the right track! Could you send me a sample repo with firebase or (roughly) the steps you took to configure/make it, I will try to replicate your environment and see why firebase isn't liking this new version of Cloud Tasks. Thanks in advance!

[nederev]

I could not reproduce it with firebase and cloud tasks 5.0.0 in minimal setup

check this rep: https://github.com/olegdater/firebase-cloud-tasks-bug (functions dir)

the deployed function testFunction from index works just fine, you can run it here: https://testfunction-xk6xc74hka-uc.a.run.app/

[sofisl]

Could you send me your full package.json (curious about the dependenices specifically...)

[sofisl]

FWIW, I also deployed my own firebase cloud function and attempted to run tasks, it worked fine as well...

[nederev]

full package.json

  "engines": {
    "node": "20"
  },
  "main": "lib/index.js",
  "dependencies": {
    "@google-cloud/aiplatform": "^3.10.0",
    "@google-cloud/bigquery": "^7.3.0",
    "@google-cloud/tasks": "4.0.0",
    "@google-cloud/vertexai": "^0.2.1",
    "airtable": "^0.12.2",
    "axios": "^1.6.7",
    "firebase-admin": "^11.11.1",
    "firebase-functions": "^4.6.0",
    "moment": "^2.29.4",
    "openai": "^4.22.0",
    "simply-beautiful": "^0.2.14",
    "zod": "^3.22.4"
  },
  "devDependencies": {
    "@types/js-beautify": "^1.14.3",
    "@typescript-eslint/eslint-plugin": "^5.62.0",
    "@typescript-eslint/parser": "^5.62.0",
    "eslint": "^8.9.0",
    "eslint-config-google": "^0.14.0",
    "eslint-plugin-import": "^2.28.1",
    "fast-csv": "^4.3.6",
    "firebase-functions-test": "^3.1.0",
    "firestore-rules-organizer": "^1.0.11",
    "ts-node": "^10.9.2",
    "tsconfig-paths": "^4.2.0",
    "typescript": "^4.9.0"
  },

[sofisl]

I looked through your dependencies to see if there were any discrepancies with Cloud tasks dependencies but couldn't find any. Just confirming, do you have a package-lock, and have you tried deleting it and reinstalling cloud-tasks at its latest version and re-installing dependencies?

If you or @LeandroTorresSicilia like to send me a more reliable reproduction of your environment, I'd be happy to take a look again.

[nederev]

did what you asked:

• rm -rf node_modules
• rm package-lock.json
• npm i

5.0.0 and 4.1.0 gives me same error even on emulator

reverting to 4.0.0 - error is gone

[nederev]

@sofisl how do I send you my full setup more privately than here?)

[sofisl]

You could add me as a collaborator for a private repo, but apparently it will give me write access, so up to you...

Ideally though, you could provide a really minimal reproduction of the error. Maybe you could fork your repo, and pare it down/delete it to the most basic reproduction.

I also tried testing it with emulators as well: https://github.com/sofisl/debug-tasks-firebase-emulator/tree/main

as well as deploying it to firebase functions: https://github.com/sofisl/debug-firebase-tasks/blob/main/src/index.ts

No luck, both work fine :/

[nederev]

will do, give me a couple of days to get to it :)

[sofisl]

@olegdater, I was never able to fully recreate your issue but I did do some work on another two issues that I was able to reproduce. Out of curiosity, could I trouble you to try 5.1.1, and see if it resolves your issue?

[kpmckellar]

I was having this issue as well with Firebase Functions and TypeScript, ended up posting on Stack Overflow and someone linked to this issue. Thought I'd share incase anything in this post helps you troubleshoot. Dropping "@google-cloud/tasks" to "4.0.1" seems to have worked for me—I am able to import CloudTasksClient without the error, https://stackoverflow.com/questions/78051822/cloudtasksclient-causing-error-when-instantiated-in-firebase-functions-ondocume

[sofisl]

@kpmckellar, would you mind giving 5.1.1 a shot, to see if it resolves your issue?

[nederev]

@sofisl just upgraded to > "@google-cloud/tasks": "5.1.1",

and even on emulator i started to get those errors again:

==
  functions: Beginning execution of "us-central1-chatGameUpdated"
>  {"severity":"ERROR","message":"Error: Failed to decode protobuf and create a before snapshot.\n   
⚠  functions: TypeError: Cannot read properties of undefined (reading 'cloud')
==

reverting back to >"@google-cloud/tasks": "4.0.0", solves the problem

[sofisl]

Bummer. Well, if anyone can send me a quick sample repo that can reproduce the problem, it would greatly be appreciated.

[nederev]

i will get to it sometime, sorry been busy, i remember :)

[motorro]

@sofisl, here's a project you may try. It uses firebase-openai-chat dependency with @google-cloud/tasks peer dependency. The client is imported here These functions fail:

• onChatCreated
• onChatUpdated

I've sent you a collaborator invite for your convenience. If you need an access to firebase project, please tell me.

Works with "@google-cloud/tasks": "4.0.1"

[sofisl]

Hi @motorro,

I attempted to reproduce with the Firebase directory, but I was unable to do so. For what it's worth, I also brought in a colleague to confirm (@danielbankhead), and we were unable to reproduce it together.

It seems like it's a good smell test that something else is going on, perhaps with the Client side, that's causing this issue. In the meantime, I'll forward to the firebase-functions team to see if they have any ideas.

For what it's worth, I followed the instructions in the README, and then deployed the functions both to gen1 and gen2 cloud functions, and called the functions from the UI and from the CLI. I even modified the code to also instantiate the Cloud Tasks in the function itself, and I was successful:

If you're interested in continuing to investigate, the next request I'd have is:

1. If you just deploy the Firebase functions from your project (i.e., just https://github.com/motorro/firebase-openai-chat-project/tree/master/Firebase), is it successful for you or do you still see the error?
2. Please send me instructions on how to build the Client to see if we can catch the error in an end-to-end reproduction.

[motorro]

Hi @sofisl @danielbankhead

Thank you for your efforts! I think there is no need to build a client. I've prepared a "digested" project that could help to reproduce the error. Cloud tasks client is used in FirebaseQueueTaskScheduler I've written a reproduction steps in a README there. Basically, you need to do the following:

1. Create Firebase project
2. Add Auth
3. Add Firestore
4. Add Functions
5. Clone the repository
6. Deploy to firebase
7. Go to Firebase console
8. Go to Firestore
9. Create a document in calculations collection
10. Check logs for onCalcCreated function

Please take a look. If you need an access to the project itself - please tell me

PS: Firestore is created at eur3 region if it helps

[sofisl]

Great, I've been able to reproduce the error! @motorro, thank you so much for your patience!

I don't have a fix yet, but I have noticed the error only occurs when triggering it from Firestore as opposed to calling it from the CLI:

The last two calls were done from the CLI.

I'll update once I have a fix!

[nederev]

wo ho! finally @sofisl :) reproduced bug - almost fixed bug :)