firebase-functions

firebase-functions copied to clipboard

[REQUIRED] Describe your environment

• Operating System version: Windows 10
• Browser version: Chrome 84
• Firebase SDK version: 7.16.1
• Firebase Product: database

[REQUIRED] Describe the problem

It's currently not possible to access the data snapshot reference when using RTDB via firebase functions.

I believe this is due to the check in https://github.com/firebase/firebase-js-sdk/blob/6af4c27743372ba531e8ce3d046ae2f81e8f5be1/packages/database/src/core/util/libs/parser.ts#L83-L91

That piece of code checks that parsedUrl.domain is localhost, but it seems to be emulator-test-1.localhost instead.

Steps to reproduce:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

You should see the following error log in the console:

i  functions: Beginning execution of "postsPushHandler"
>  [2020-07-22T15:49:53.227Z]  @firebase/database: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
!  functions: Error: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com
    at fatal (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:341:11)
    at parseRepoInfo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:1296:9)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:14990:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at DataSnapshot.get ref [as ref] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\providers\database.js:293:34)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:17:24
    at cloudFunction (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-functions\lib\cloud-functions.js:132:23)
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:573:20
!  Your function was killed because it raised an unhandled error.

Relevant Code:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
  console.log(snapshot.ref)
});

cc @samtstern (because I've reported other emulator related issues to you)

@filipesilva what version of firebase-functions are you using? This should be fixed in version 3.8.0 (assuming newest version of the CLI as well)

The repro (https://github.com/filipesilva/firebase-emulator-parse-url) is using [email protected]. Everything there is the latest as of today, I believe.

@filipesilva sorry I should have clicked into that

Ok a few things here:

1. I was able to reproduce this
2. If I change the initialization block to just admin.initializeApp() this error goes away.

It looks like parseRepoInfo() is being called with a URL like:

https://emulator-test-1.localhost

This is a firebase-functions issue, transferring.

Ok so I added some logging and extractInstanceAndPath is being called like this:

>  extractInstanceAndPath(projects/_/instances/emulator-test-1/refs/posts/-MCvh_klBc9LYP7JIlsJ, localhost)
>  return [https://emulator-test-1.localhost,/posts/-MCvh_klBc9LYP7JIlsJ]

The dataConstructor in onCreate is being given this raw.context:

{
  "eventType": "google.firebase.database.ref.create",
  "params": {
    "postId": "-MCviHJhohdsNE5uAJWZ"
  },
  "domain": "localhost",
  "resource": {
    "service": "firebaseio.com",
    "name": "projects/_/instances/emulator-test-1/refs/posts/-MCviHJhohdsNE5uAJWZ"
  },
  "timestamp": "2020-07-23T12:49:58.150Z",
  "eventId": "5/3OSuz1TP5pWvc/Lt0o9RDcN2M=",
  "authType": "ADMIN"
}

So the port is lost ... gonna have to think about this one.

@filipesilva question for you: when you're running inside the Functions emulator and you also have the RTDB emulator running, but you do this:

admin.initializeApp({
   databaseURL: "https://emulator-test-1.firebaseio.com/",
   credential: admin.credential.applicationDefault()
});

const db = admin.database()

Do you still expect db to point to the emulator? To me it looks like you're explicitly trying to access production, otherwise you could do admin.initializeApp() to accept the defaults.

So I'm trying to decide if the actual bug is that your posts function writes to the emulators at all.

@samtstern I don't quite care about setting credential and would be happy enough if that is just defaulted.

I do care about setting databaseURL because I test a sharded RTDB and it is important that functions I have can be routed to the correct shard, and that the shard is using the emulators. I think this is what happens today because, in my particular work setup, when I save security rules, it will update rules for all 100 shards I use, and they show up in the emulator UI. I could use a URL that's pointing to the emulators by encoding that as a build time variable.

I also care about setting databaseAuthVariableOverride as well because I want my writes from firebase functions to still obey validation rules.

Attempting to set it with no databaseURL (e.g. admin.initializeApp({databaseAuthVariableOverride: "something"}); ) results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Can't determine Firebase Database URL.
    at FirebaseDatabaseError.FirebaseError [as constructor] (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:43:28)       
    at new FirebaseDatabaseError (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\utils\error.js:204:23)
    at DatabaseService.ensureUrl (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:89:15)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:56:26)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:8:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
    at Generator.next (<anonymous>)
!  Your function was killed because it raised an unhandled error.

Attempting to set it with databaseURL but no credential also results in a runtime error:

i  functions: Beginning execution of "posts"
!  functions: Error: Only objects are supported for option databaseAuthVariableOverride
    at new Repo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:12513:27)
    at RepoManager.createRepo (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15049:16)
    at RepoManager.databaseFromApp (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15014:25)
    at Object.initStandalone (D:\sandbox\firebase-emulator-parse-url\node_modules\@firebase\database\dist\index.node.cjs.js:15389:45)
    at DatabaseService.getDatabase (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\database\database.js:67:23)
    at FirebaseApp.database (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-app.js:232:24)
    at Proxy.fn (D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-admin\lib\firebase-namespace.js:280:45)
    at D:\sandbox\firebase-emulator-parse-url\functions\index.js:10:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:20
    at D:\sandbox\firebase-emulator-parse-url\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:558:19
!  Your function was killed because it raised an unhandled error.

So in my concrete work example it does not seem possible to just go with the defaults. I must set credential and databaseURL in order to set databaseAuthVariableOverride. In some cases I must also set databaseURL to access shards.

@filipesilva thanks for clarifying all of that ... there's a lot going on here and I'll have to think of the best way to solve it.

@samtstern heya, did you have time to think about what a resolution for this looks like?

@filipesilva thanks for bumping this issue! I took another look today and think I found a really simple fix: https://github.com/firebase/firebase-functions/pull/838

Here are the functions I am using:

const functions = require('firebase-functions');
const admin = require('firebase-admin');

const instance = "fir-dumpster-secondary";

admin.initializeApp({
  databaseURL: `https://${instance}.firebaseio.com/`,
  credential: admin.credential.applicationDefault()
});

exports.posts = functions.https.onRequest(async (request, response) => {
  const db = admin.database();
  const ref = await db.ref('posts').push({
    date: new Date().toISOString()
  });
  response.send(`Added: ${ref}`);
});

exports.postsPushHandler = functions.database
  .instance(instance)
  .ref('/posts/{postId}')
  .onCreate(snapshot => {
    console.log("onCreate:", snapshot.ref.toString());
    return true;
  });

And here are the logs I get:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  onCreate: http://localhost:9000/posts/-MPslO6MOO53iuSWFUJL
i  functions: Finished "postsPushHandler" in ~1s

In the Emulator UI it all seems wired up correctly:

Would you mind testing this for me? You can check out my branch of this repo and then run npm run build. Then in your functions repo run npm install --save /path/to/your/clone/firebase-functions which should let you use the local version.

@samtstern thanks for getting back to me so quickly! I tested your branch build on https://github.com/filipesilva/firebase-emulator-parse-url and can confirm I no longer get the URL error, and that the write correctly goes through. I think that fixes it!

@samtstern today I was integrating your changes into our codebase, and I found a follow up problem with the fix you submitted.

I've updated the original repository with the most recent firebase versions and code to repro this problem in https://github.com/filipesilva/firebase-emulator-parse-url/commit/979d5c65808b15a423ee29aff0c33748c3c9a9eb.

The repro steps are still the same:

git clone https://github.com/filipesilva/firebase-emulator-parse-url
cd firebase-emulator-parse-url
yarn
yarn emulators
# open http://localhost:5001/emulator-test-1/us-central1/posts in the browser

This time the output is:

i  functions: Beginning execution of "posts"
i  functions: Finished "posts" in ~1s
i  functions: Beginning execution of "postsPushHandler"
>  snapshot ref http://localhost:9000/posts/-MXYBOwrxE0SXK0MOAmN
>  admin app options {
>    databaseURL: 'https://emulator-test-1.firebaseio.com/',
>    credential: RefreshTokenCredential {
>      httpAgent: undefined,
>      implicit: true,
>      refreshToken: RefreshToken {
>        clientId: '563584335869-fgrhgmd47bqnekij5i8b5pr03ho849e6.apps.googleusercontent.com',
>        clientSecret: 'j9iVZfS8kkCEFUPaAeJV0sAi',
>        refreshToken: '1//0dffIHm3tl_W7CgYIARAAGA0SNwF-L9IrkirUzBA22L9-rvZHCDa4jcM47eAhdK2USE-8miFIGJkYRn39CoFzhFTErl82HejNLJU',
>        type: 'authorized_user'
>      },
>      httpClient: HttpClient { retry: [Object] }
>    }
>  }
>  [2021-04-05T18:00:42.083Z]  @firebase/database: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
⚠  functions: Error: FIREBASE FATAL ERROR: Database initialized multiple times. Please make sure the format of the database URL matches with each database() call.
    at fatal (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:341:11)
    at RepoManager.createRepo (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15218:13)
    at RepoManager.databaseFromApp (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15185:25)
    at initStandalone (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15462:45)
    at Object.initStandalone$1 [as initStandalone] (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/node_modules/@firebase/database/dist/index.node.cjs.js:15594:12)
    at DatabaseService.getDatabase (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/database/database-internal.js:80:23)
    at FirebaseApp.database (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-admin/lib/firebase-app.js:158:24)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/functions/index.js:19:40
    at cloudFunction (/Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-functions/lib/cloud-functions.js:134:23)
    at /Users/filipesilva/sandbox/firebase-emulator-parse-url/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:592:16
⚠  Your function was killed because it raised an unhandled error.

This logging comes from the updated handler:

exports.postsPushHandler = functions.database.ref('/posts/{postId}').onCreate(snapshot => {
   console.log("snapshot ref", snapshot.ref.toString())
   console.log("admin app options", adminApp.options)
   console.log("admin app db", adminApp.database())
 });

Attempting to access the database for the initialised admin app results in an error. There is no error if console.log("snapshot ref", snapshot.ref.toString()) is commented out.

It seems that using the snapshot ref initialises a database for the same url, but with a different URL.

Can this issue be reopened, or should I open a new issue?

@filipesilva sure let's reopen this issue. So from that commit you sent it looks like there's a regression in one of the recent SDK updates. Could be firebase-admin, could be firebase-functions, etc. Do you think you could help me narrow it down by figuring out which of those changes in your commit leads to the regression?

Hi @samtstern! I tried going back to the original package versions for everything but firebase-functions in https://github.com/filipesilva/firebase-emulator-parse-url/commit/9ea2e64603b4f4d123fb248cce2cff012e964f95. The new problem still repros so I think it's related to firebase-functions.

I'm not even sure it's a regression though. I didn't test this originally with your fix because it required a larger time investment in updating our codebase, so it's possible that it was never working either.