🐛 [firestore-bigquery-export] Root collection wildcard regression

[kdawgwilk]

Configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.2.2
• Configuration values (redact info where appropriate):
  • collection: {collection}

Steps to reproduce:

1. Install the extension while trying to use a global collection wildcard {collection}

Expected result

Extension deploys successfully

Actual result

Extension deploy fails with the following error

failed to startDeploymentManagerWork for instance id: firestore-bigquery-export, parentId: my-project-id: error updating deployment for "projects/my-project-id/global/deployments/firestore-bigquery-export" with delete policy "ABANDON": generic::internal: DM HTTP 412 error. Body: { "error": { "code": 412, "message": "Error parsing configuration: while parsing a block mapping\n in 'string', line 71, column 19:\n - attribute: document\n ^\nexpected \u003cblock end\u003e, but found '\u003cscalar\u003e'\n in 'string', line 73, column 38:\n ... value: {collection}/{documentId}\n ^\n", "errors": [ { "message": "Error parsing configuration: while parsing a block mapping\n in 'string', line 71, column 19:\n - attribute: document\n ^\nexpected \u003cblock end\u003e, but found '\u003cscalar\u003e'\n in 'string', line 73, column 38:\n ... value: {collection}/{documentId}\n ^\n", "domain": "global", "reason": "conditionNotMet", "location": "If-Match", "locationType": "header" } ] } }

This was previously working as expected before i updated to 0.2.2

[cabljac]

Hi, thanks for raising this, i'll look into it and provide updates ASAP.

[cabljac]

Can you let me know which version it worked on?

[cabljac]

I believe this is a deployment manager issue, and we can resolve it in our extension yaml maybe with

•        value: ${COLLECTION_PATH}/{documentId}
  

•        value: '${COLLECTION_PATH}/{documentId}'
  

I'll prep a PR and keep you updated

[cabljac]

hm after experimenting with a bunch of different solutions here, it might be a platform issue with the new triggers. I will raise this with the team and get back to you.

[kdawgwilk]

Versions < 0.2.0 worked for me

[CorieW]

Notes after some testing:

• Confirmed that it worked in v0.1.x.
• v0.2.x fails to deploy.
• Non-root wildcards work fine in v0.2.x.
• v2 triggers support wildcards by my testing, so this is odd behaviour.

[cabljac]

\u003cscalar\u003e is <scalar> by the way, interesting

[cabljac]

Hi there, this seems to be a problem with the extensions backend. We've raised it with the team and will provide updates as we get them. In the meantime we have a workaround which should allow root wildcards in the latest version of the extension:

Workaround

:warning: Disclaimer: proceed with caution here, I don't know what will happen if you try to reconfigure the extension through the Firebase Console after following this workaround.

1. install your extension as you have done but just with a placeholder for the Collection path param:

:warning: It's actually worth choosing a value that warns this is modified manually elsewhere, or something as the displayed param value in the firebase console will NOT change after applying this workaround

The other params can be set to whatever you want, for example if you want to extract path params, have that enabled.

2. Navigate in the google cloud console to the function revision. It will be at:

https://console.cloud.google.com/run/detail/<LOCATION>/ext-<EXT_INSTANCE_ID>-fsexportbigquery/revisions?project=<PROJECT_ID>

Where you should substitute in LOCATION, PROJECT_ID, EXT_INSTANCE_ID as your configured function location, GCP project ID, and extension instance ID (default is firestore-bigquery-export, but you may have renamed this during installation)

Or via the UI go to Cloud Run Functions

\ **and then click on the function**

Navigate to Triggers:

Then create a new trigger by clicking Add Trigger:

Select "Other Eventarc Trigger"

**Pick Cloud Firestore as the source and google.cloud.firestore.document.v1.written as the event type.

Make sure you set the Region to the same region of your Cloud Firestore instance, and the database filter should also match your extension configuration.**

Importantly add a filter with Attribute document and operator Path pattern and then Attribute value must be {collection}/{documentId}

Here is an example:

Once you've added this trigger, you will want to delete the existing trigger to prevent duplicate data in your tables.

Finally if you want to extract path params from the documents you will need to modify the COLLECTION_PATH in the env variables of the deployed function, by deploying a new revision. For this:

1. go back to the function page 2.Edit & deploy a new revision
2. under "Edit Container" find "Variables & Secrets"
3. set COLLECTION_PATH to {collection}/{documentId}
4. click Deploy

Appreciate this is quite a long winded workaround. As i said, i will provide updates on the backend fix as soon as I get them.