firebase
π [Stream Firestore to BigQuery] fs-bq-import-collection doesn't recognize partitioned field
[REQUIRED] Step 2: Describe your configuration
- Extension name: firestore-bigquery-export (
storage-resize-images,firestore-send-email, etc) - Extension version: 0.1.49
[REQUIRED] Step 3: Describe the problem
When I install extension with following configuration, it creates BigQuery table with createdAt partitioned field.
The important thing is its extension project id and BigQuery id is different (Cross-Platform BigQuery).
But When I run fs-bq-import-collection to backfill, it doesn't recognize any partitioned fields. The script shows following debug message.
{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}
Steps to reproduce:
Create Firestore table with createdAt TIMESTAMP field and Install extension with above configuration.
Expected result
fs-bq-import-collection recognize createdAt partition field and set it when importing.
Actual result
fs-bq-import-collection doesn't recognize createdAt partition field and set null to createAt field when importing.
From the POSTINSATALL:
Note: To enable partitioning for a Big Query database, the following fields are required:
- Time Partitioning option type
- Time partitioning column name
- Time partiitioning table schema
- Firestore document field name
`Clustering` will not need to create or modify a table when adding clustering options, this will be updated automatically.
You're expecting the table to be created with partitioning on createdAt, however the configuration doesn't specify that as a Time partitioning column name, the one you have is timestamp.
Having the same issue. The bigquery table is partitioned (see below) but the following warning is thrown at the start of fs-bq-import-collection.
{"severity":"INFO","message":"BigQuery table with name users_raw_changelog already exists in dataset ***!"}
{"severity":"INFO","message":"Clustering removed on users_raw_changelog"}
{"severity":"WARNING","message":"Did not add partitioning to schema: Partitioning not enabled"}
Table Type Partitioned Partitioned by DAY Partitioned on field creation_timestamp Partition expiration Partitions do not expire Partition filter Not required
I think the OP's issue is misconfiguration of the extension, as @pr-Mais mentions.
@KristofSochan are you also using the script cross-project, or from firestore to bigquery on the same GCP project?
Can you provide:
a) your extension configuration b) the import script command you are using
