firebase
🐛 [firestore-bigquery-export] Missing rows in bq after export
[REQUIRED] Step 2: Describe your configuration
Stream Firestore to BigQuery firebase/[email protected]
BIGQUERY_PROJECT_ID=*** COLLECTION_PATH=*** DATASET_ID=*** DATASET_LOCATION=eu firebaseextensions.v1beta.function/location=europe-west1 TABLE_ID=*** TABLE_PARTITIONING=NONE TIME_PARTITIONING_FIELD_TYPE=omit USE_NEW_SNAPSHOT_QUERY_SYNTAX=yes WILDCARD_IDS=false%
[REQUIRED] Step 3: Describe the problem
Install the extension with the latest version and configure. Insert 2_000_000 events in firebase (in chunks of 500). Takes about 20 minutes. Wait for bq table to be filled.
Expected result
Rows count in bq table to be 2,000,000.
Actual result
Rows in bq are: 1,977,677. Functions execution stopped before 2 hours (seen in logs).
Logs show the following:
"Function execution started" filter applied resource.labels.function_name="ext-firestore-bigquery-export-syncBigQuery - 1,977,762 log entries resource.labels.function_name="ext-firestore-bigquery-export-fsexportbigquery - 2,000,115 log entries
So it seems that probably syncBigQuery failed to start for some reason at some point. I didn't see any errors in logs explorer that could show the cause.
Any help would be appreciated.
Hi there, thanks for raising this, I will investigate this. In the meantime, if it helps, we have a Backup Collection Name parameter - You could set this and hopefully it should catch the failed documents, so you know which havent been exported
Thanks for the quick response. I'll add the Backup Collection functionality to check whether this helps.
Hi,
is there any update on this? we're also experiencing missing updates on the bigquery side.
Hi,
is this issue persisting in versions 0.1.56+ of the extension? We made several updates to improve data consistency since 0.1.41.
Closing as I believe this was resolved in 0.1.56+, feel free to reopen if the issues persists.