🐛 [firestore-bigquery-export] Collection linkage fails

[tt-ryosuke-kikuchi]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export(Stream Firestore to BigQuery)
• Extension version: 0.1.35
• Configuration values (redact info where appropriate):
  • _
  • _

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

First, link samplecollections to BigQuery using 'Stream Firestore to BigQuery' from Extensions on FirebaseConsole.

After that, I used fs-bq-import-collection to synchronize the Firestore data.

The following error occurred and synchronization was not possible.

This is a log of execution details and results.

MacBook-Pro:workspace $ npx @firebaseextensions/fs-bq-import-collection \
>   --non-interactive \
>   --project=sample-project \
>   --source-collection-path=samplecollections \
>   --query-collection-group=false \
>   --dataset=firestore_export \
>   --table-name-prefix=samplecollections \
>   --batch-size=300 \
>   --dataset-location= \
>   --multi-threaded=false \
>   --use-new-snapshot-query-syntax=false \
>   --use-emulator=false
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
Importing data from Cloud Firestore Collection: samplecollections, to BigQuery Dataset: firestore_export, Table: samplecollections_raw_changelog
[Error: ENOENT: no such file or directory, unlink '/Users/user/.npm/_npx/d906c8461d95e111/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-samplecollections-to-sample-project_firestore_export_samplecollections_raw_changelog'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'unlink',
  path: '/Users/user/.npm/_npx/d906c8461d95e111/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-samplecollections-to-sample-project_firestore_export_samplecollections_raw_changelog'
}
Error unlinking journal file /Users/user/.npm/_npx/d906c8461d95e111/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-samplecollections-to-sample-project_firestore_export_samplecollections_raw_changelog after successful import: Error: ENOENT: no such file or directory, unlink '/Users/user/.npm/_npx/d906c8461d95e111/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-samplecollections-to-sample-project_firestore_export_samplecollections_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------

Expected result

First, when 'Stream Firestore to BigQuery' is set, the data at that time will be synchronized on BigQuery.

After that, running fs-bq-import-collection will keep the data in sync.

Actual result

In reality, when you set 'Stream Firestore to BigQuery', the data at that point is not synchronized on BigQuery, and the content of the table created on BigQuery is 0Kbyte.

Then running fs-bq-import-collection also fails.

It seems that not all synchronizations fail, but only certain types of collections.

Successful collections have the following format:

collections 
  key: [
    key:value,
    key:value
  ]

The format of collections that fail is as follows:

collections 
  key: [
    collectionsB {
      key:value,
      key:value
    },
    collectionsB {
      key:value,
      key:value
    }
  ]

I'm guessing that it probably doesn't work unless there is a collection directly under the key.

Thank you for your support.

[cabljac]

Hi there, could you post your extension config parameters so i can try and reproduce? Also would it be possible to send a screenshot of examples of collections that do and do not fail, as I'm a bit confused by the formats you gave

[cabljac]

Hi, i'm going to close this as stale. Feel free to reopen if you're able to provide clarifying information.