firebase
🐛 [Stream Firestore to BigQuery] Unable to stream Firestore DB to a different project
[READ] Step 1: Are you in the right place?
Yes, for Stream Firestore to BigQuery extension, error during setup.
[REQUIRED] Step 2: Describe your configuration
- Extension name: Stream Firestore to BigQuery
- Extension version: 0.1.52
- Configuration values (redact info where appropriate):
- BigQuery Dataset location:
us - BigQuery Project ID:
project-name(Different than location of Firestore DB) - Collection path:
users - Enable Wildcard Column field with Parent Firestore Document IDs (Optional):
false - Dataset ID:
dataset-name - Table ID:
users - BigQuery SQL table Time Partitioning option type (Optional):
DAY - BigQuery Time Partitioning column name (Optional):
creation_timestamp - Firestore Document field name for BigQuery SQL Time Partitioning field option (Optional):
creation_timestamp - BigQuery SQL Time Partitioning table schema field(column) type (Optional):
TIMESTAMP - BigQuery SQL table clustering (Optional):
Parameter not set - Maximum number of synced documents per second (Optional):
100 - Backup Collection Name (Optional):
Parameter not set - Transform function URL (Optional):
Parameter not set - Use new query syntax for snapshots:
yes - Exclude old data payloads (Optional):
yes - Use Collection Group query (Optional):
no - Cloud KMS key name (Optional):
Parameter not set
- BigQuery Dataset location:
[REQUIRED] Step 3: Describe the problem
The extension is stuck in "Configuring BigQuery Sync" when I check "Runtime status" in the extension management page.
When I check the cloud functions that were triggered, I see this error on the function: syncBigQuery
ext-firestore-bigquery-export-users2-syncBigQueryqu3g9rrcqx5t Unhandled error ApiError: Access Denied: Dataset project-name:dataset-name: Permission bigquery.datasets.get denied on dataset project-name:dataset-name (or it may not exist). at new ApiError (/workspace/node_modules/@google-cloud/common/build/src/util.js:75:15) at Util.parseHttpRespBody (/workspace/node_modules/@google-cloud/common/build/src/util.js:210:38) at Util.handleResp (/workspace/node_modules/@google-cloud/common/build/src/util.js:151:117) at /workspace/node_modules/@google-cloud/common/build/src/util.js:534:22 at onResponse (/workspace/node_modules/@google-cloud/common/node_modules/retry-request/index.js:259:7) at /workspace/node_modules/teeny-request/build/src/index.js:226:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { code: 403, errors: [ { message: 'Access Denied: Dataset project-name:dataset-name: Permission bigquery.datasets.get denied on dataset project-name:dataset-name (or it may not exist).', domain: 'global', reason: 'accessDenied' } ],
Steps to reproduce:
Configure Big Query streaming from Firestore with the above configuration, BigQuery Project ID must be different than the Firestore source project.
I changed the dataset name to something else other than the default (which was firestore_export).
Expected result
Extension is properly initialized, i.e. Runtime status should say "Sync setup completed".
Actual result
Extension is not properly initialized due to permission errors across project, see above.
The info for the "Dataset ID" field in the extension configuration says: "What ID would you like to use for your BigQuery dataset? This extension will create the dataset, if it doesn’t already exist."
Unsure how else to allow the extension to create a dataset in a different project.
So the permission needs to be manually added after installation of the extension. I've opened a PR which adds bash/powershell scripts to assist in this, and added documentation to guide you through it.