🐛 [firestore-bigquery-export] Streaming is not working properly

[aya-z-arketa]

Describe your configuration

• Extension name: firestore-bigquery-export-service-cat
• Extension version: firebase/[email protected]
• Configuration values (redact info where appropriate):
  • Firestore Project ID: sutra-prod
  • BigQuery Project ID: sutra-prod
  • Firestore Instance Location: nam5
  • Collection path: users/{userId}/serviceCategories
  • Enable Wildcard Column field with Parent Firestore Document IDs (Optional): true
  • Dataset ID: firestore_export_service_categories
  • Table ID: service_categories

everything else was kept as default values

Describe the problem

Steps to reproduce:

I created a new document in one of the collections and it was not streamed into bigquery. I took this loom video to show the problem. Apologies for the background noise.

Expected result

The document should appear in my bigquery export within seconds.

Actual result

The document does not get sent to bigquery.

[aya-z-arketa]

@cabljac let me know if I can provide any additional info! I downgraded to 0.1.58 and it works great.

[cabljac]

Thanks! I'll investigate further and keep you updated!

[cabljac]

Hi, if you go into the logs of the cloud function managed by the extension, do you see any errors?

I've found with the new v2 functions that it is difficult to actually find these logs, i think another issue with the platform.

if you go to the cloud run page for the function in GCP console, something like this:

https://console.cloud.google.com/run/detail/<FUNCTION_LOCATION>/ext-<EXTENSION_INSTANCE_ID>-fsexportbigquery/logs

(make sure you're in the correct GCP project)

Or you can click on view logs here for fsbigqueryexport

and then copy and paste the function id in the search

[aya-z-arketa]

I didn't see any errors, not sure what's happening. But downgrading resolved it

[erikjernstrom]

We have same issue, i.e. that replication is not working after upgrading extension version. We don't want to downgrade since we need to be able to have separate FireStore DataBase instead of default, so waiting for the extension to work.

The only thing I see in the logs of the function "fsexportbigquery" are these warnings: "The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header. Read more at https://cloud.google.com/run/docs/securing/authenticating Additional troubleshooting documentation can be found at: https://cloud.google.com/run/docs/troubleshooting#unauthorized-client"

We have tried with latest extension version, i.e. 0.2.4.

[cabljac]

ah interesting @erikjernstrom thanks for additional context. I'll provide updates after further investigation

[ushiromuki]

We are experiencing the same issue on version 0.2.4.

The logs show the following warning: “The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header,” and we are unable to export from Firestore to BigQuery.

Could this be because the required role, such as run.invoker, is missing from the extension.yaml?

[htsuruo]

We've seen a similar issue, although not with this extension.

As you know, the default service account for Eventarc, which is used by 2nd generation Cloud Functions (Cloud Run functions), is automatically attached [PROJECT_NUMBER][email protected]. However, this service account is not granted the Cloud Run Invoker role (run.invoker). Therefore, Cloud Run functions via Firebase Extensions does not have permission to invoke Cloud Run functions and output following error logs as default.

The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header

We can resolve the issue by adding run.invoker role the service account manually, but is there any ways for the Extensions to manage/control this automatically?

[cabljac]

Ah @htsuruo that's very useful info, I'll look into this ASAP. In theory extensions should be granting this role automatically, but perhaps with the v2 update something isn't happening

[htsuruo]

@cabljac

Thank you your reply.

In theory extensions should be granting this role automatically

I'm interested in where the code written. Could you tell me about this? And ideally, the extension tool should create new service account for the Eventarc instead of default service account the following best practice.

[cabljac]

I'm interested in where the code written. Could you tell me about this?

So some roles are granted to the extension by the extensions platform, and other roles we can add to the extension.yaml. This is not a role that we add explicitly but one that should be granted by the platform, as far as I understand.

And ideally, the extension tool should create new service account for the Eventarc instead of default service account the following best practice.

Can you file a new issue for this please?

[cabljac]

We've seen a similar issue, although not with this extension.

Which extension did you see this with? @htsuruo

[htsuruo]

@cabljac

I’m actually an extensions developer myself, and the extension I found the issue is mine. https://extensions.dev/extensions/htsuruo/trigger-github-issues-from-crashlytics

As you know, Firebase Alerts only supported Cloud Run functions(2nd-gen Cloud Functions). As a user reported an issue, I looked into and install the extension in blank Firebase project. After that, I faced on the issue not enough permission to invoke Cloud Run functions. Probably, the cause may be something to update packages because I run ‘npm upgrade’ in development. Not sure.

You may be able to reproduce the issue by installing my extension from following pre-release link. https://console.firebase.google.com/project/_/extensions/install?ref=htsuruo/[email protected]

I’m happy to help you.

[cabljac]

Ah yes I've seen this extension @htsuruo, cool!

Can you open a fresh issue for the missing role? I don't know if it's related to this issue. I'll meet with the Firebase team tomorrow, and can bring this up with them.

note: I haven't been able to reproduce this yet, still trying

[cabljac]

Well it seems related, but worth tracking both

[htsuruo]

@cabljac

Can you open a fresh issue for the missing role?

Sure. I’ll do that.

I'll meet with the Firebase team tomorrow, and can bring this up with them.

Cool! I’m looking forward to finding the cause.

Additional info: My extension has been working well since its initial release two years ago. However, I recently developed it further, and an issue(missing role) has come up.

[htsuruo]

@cabljac

I found related code in firebase-tools repository. https://github.com/firebase/firebase-tools/blob/629f211c35032a2dea3343e408858b1043143af9/src/deploy/extensions/v2FunctionHelper.ts#L10

The missing role issue is not specific but common one by using 2nd-gen functions. So, I'll report the issue on firebase-tools repo instead of this one.

Appreciate it.

[cabljac]

So i think the invoker role needs to be granted to the allUsers principal on the function in question. For 1st gen functions, this is roles/cloudfunctions.invoker; for 2nd gen (Cloud Run-based) functions, it's roles/run.invoker.

It might be that organization-level policies or allowlists might prevent this role from being granted, and extensions may not have sufficient permission to configure this access automatically.

I will try to provide further updates, thanks everyone for you patience!

[htsuruo]

@cabljac

Thank you asking firebase team about the issue. However, I don't set up any organization policies and allowlists. So, It might be not main cause.

Anyway, the discussion is far from this issue, could you discuss about missing role issue on https://github.com/firebase/firebase-tools/issues/8629 ?

[cabljac]

@htsuruo is the problem still occurring for you? There have been some platform changes since this issue was opened.

[aya-z-arketa]

@cabljac I just upgraded to 0.2.5 and this is still happening. I do not see any errors in the logs.

[erikjernstrom]

@cabljac same here, can you please add the missing role so Cloud Function v2 works for this extension? We need this extension to work for a go-live later this year. Our fallback solution is to downgrade and use the default FireStore database, but we would like to avoid that if possible.

[cabljac]

Hi all, we will reinvestigate this - it's proven difficult for us to reproduce, but I will provide more updates as soon as I can

[cabljac]

@aya-z-arketa thanks for reproducing by the way!

It might be worth raising a support ticket and seeing if they can do some introspection of your project and the roles available in the meantime.

[erikjernstrom]

Hi, Just so I understand, the latest version of the extension is working for you? Very confused how that is possible unless you run in a project where you are granting extra permissions to service account (either default compute or the extension one). Neither the default compute or the extension accounts have this permission, so you must have granted that in your project separately. Could you add run.invoker role to extension.yaml?

[cabljac]

Hey,

run.invoker should be being added automatically, I will test on a completely fresh project today and get back to you!

[cabljac]

UPDATE

OK so i tested pretty extensively on a fresh project and the latest extension version (0.2.5).

I think i've reproduced it now.

@erikjernstrom you were correct. New projects seem to be granting Editor to the default compute account, which includes invoker permissions.

I removed Editor, and it stopped working.

If i grant either eventReceiver or run.invoker then it seems to work.

I did notice this section when installing, which i hadn't seen before:

I'm not sure service agent is the correct service account here though - i'm continuing with testing for now to try and get to the bottom of it. Thanks all for your patience!

[cabljac]

The issue is, adding invoker to the extension.yaml won't fix things, since it's not the service account that invokes the function. It seems like it's the default compute one which is invoking.

[erikjernstrom]

Thank you!

Yes, but why is not the service account specifically created for the extension doing the invocation? I think I read that EventArc is used, and I double checked now that you can run EventArc as your service account, why aren't you?

Using Compute engine Service Account and Editor role is fine for sandbox/PoC, but we need to apply Principle of Least Access on our production flows.

Should I grant run.invoke on compute SA (workaround from my perspective) or can you try using the extensions service account for the invocation in Cloud Run? An add run.invoke to the extension.yaml.

[cabljac]

I've raised this with the extensions platform team, as I don't have direct insight into the architecture/backend.

Using Compute engine Service Account and Editor role is fine for sandbox/PoC, but we need to apply Principle of Least Access on our production flows.

I would agree, this makes sense to me!

Should I grant run.invoke on compute SA (workaround from my perspective) or can you try using the extensions service account for the invocation in Cloud Run? An add run.invoke to the extension.yaml.

For now I think the workaround is to grant the minimal required roles (run.invoke and/or the EventArc event receiver role) to the SA. I am following up with the platform team to try and resolve this.