Stream Firestore to BigQuery extension installation fails

[thandal]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firebase/[email protected]
• Extension version: 0.2.5
• Configuration values (redact info where appropriate):
  • basically the defaults, setting the firestore location to my location

[REQUIRED] Step 3: Describe the problem

When I try to install the extension, for example

https://console.firebase.google.com/project/PROJECT-460819/extensions/install?ref=firebase%[email protected]

it eventually fails:

Details look like:

; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export/resources/initBigQuerySync: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Failed to create 1st Gen function projects/PROJECT-460819/locations/us-central1/functions/ext-firestore-bigquery-export-initBigQuerySync: Default service account '[email protected]' doesn't exist. Please recreate this account or specify a different account. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.","status":"FAILED_PRECONDITION","statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/PROJECT-460819/locations/us-central1/functions","httpMethod":"POST"}}; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export/resources/syncBigQuery: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Failed to create 1st Gen function projects/PROJECT-460819/locations/us-central1/functions/ext-firestore-bigquery-export-syncBigQuery: Default service account '[email protected]' doesn't exist. Please recreate this account or specify a different account. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.","status":"FAILED_PRECONDITION","statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/PROJECT-460819/locations/us-central1/functions","httpMethod":"POST"}}; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export/resources/fsexportbigquery: {"ResourceType":"gcp-types/cloudfunctions-v2beta:projects.locations.functions","ResourceErrorCode":"404","ResourceErrorMessage":{"code":404,"message":"Service account projects/-/serviceAccounts/[email protected] was not found.","status":"NOT_FOUND","statusMessage":"Not Found","requestPath":"https://cloudfunctions.googleapis.com/v2beta/projects/PROJECT-460819/locations/us-central1/functions","httpMethod":"POST"}}; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export/resources/setupBigQuerySync: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Failed to create 1st Gen function projects/PROJECT-460819/locations/us-central1/functions/ext-firestore-bigquery-export-setupBigQuerySync: Default service account '[email protected]' doesn't exist. Please recreate this account or specify a different account. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.","status":"FAILED_PRECONDITION","statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v1/projects/PROJECT-460819/locations/us-central1/functions","httpMethod":"POST"}}

Steps to reproduce:

Something suspicious: when I click the "Grant" buttons on this page of the installation sequence,

I don't get positive feedback... the buttons just disappear:

Expected result

Extension installs cleanly

Actual result

Extension fails to install!

I have also tried to install the extension with the command line, which generates a slightly different error:

=== Deploying to 'PROJECT-460819'...

i  deploying extensions
i  extensions: ensuring required API firebaseextensions.googleapis.com is enabled...
✔  extensions: required API firebaseextensions.googleapis.com is enabled
i  extensions: ensuring required API compute.googleapis.com is enabled...
✔  extensions: required API compute.googleapis.com is enabled

Error: Request to https://cloudresourcemanager.googleapis.com/v1/projects/PROJECT-460819:setIamPolicy had HTTP Error: 400, Service account [email protected] does not exist.

I strongly suspect this is related to the following issues, but since it involved the Firebase web interface, I thought it deserved its own bug:

• https://github.com/firebase/extensions/issues/2383
• https://github.com/firebase/firebase-tools/pull/8566

[HassanBahati]

Hi @thandal, thanks for reporting this issue. We've received it and are reviewing it. We'll provide updates as soon as possible.

[MrLibya]

Yeah its in backend side, backend still looking for hard-coded service account, instead of looking up for the default service account.

[felansu]

@cabljac @thandal @HassanBahati @mbleigh @rafikhan

• Any new about the hard-coded service account in backend, was resolved?
• Is in the roadmap finish that part of backend?

[cabljac]

Hi there, I'll bring this up with the team for a fresh update on the status of this!

For extra context, I have seen this reproduced several times. I have seen it install correctly after clicking "reinstall" but not always.

[MrLibya]

For extra context, I have seen this reproduced several times. I have seen it install correctly after clicking "reinstall" but not always.

Its not about this extension, Even with any custom extension that uses V2 of cloud function and having different default compute service account will fail because when deploying the backend of gcloud extension will try to look for the hard-coded default service account instead of searching for the correct default service account.

Default compute service account is [email protected], in some projects they are deleted and can create new default service account with help of support but it will be on different name,

[felansu]

@cabljac is exactly what @MrLibya describes: the service account should be fully specifiable, not just using PROJECT_ID + [email protected].

In all other products, I’ve always specified the exact service account to use — why should this be different here?

[hernandoKoggi]

Hi, just checking if there are any updates on the issue. Thanks!

[cabljac]

Hey there, i've labelled this as a Platform issue. Thanks for the input, hopefully this will narrow it down for a fix on the backend. I'll continue to raise with the extensions team and provide updates as they come