🐛 [firestore-bigquery-export] Docs with @type keys throw error and aren't synced

[ca136]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: latest
• Configuration values (redact info where appropriate): not relevant

[REQUIRED] Step 3: Describe the problem

We recently updated our extension and noticed a bunch of new Data cannot be decoded from JSON. errors. It's not clear to me if these are showing up because the function structure or logging changed or if the actual json parsing has changed since we upgraded.

The current version of the extension appears to be using the decode method from the firebase SDK https module, which throws an error if an object has a key of @type with a value that isn't LONG_TYPE or UNSIGNED_LONG_TYPE. We use the @type field in a ton of places and it's not feasible to change the structure of our data model. Is there a way to work around the issue or somehow bypass this decoding error?

Steps to reproduce:

Create a firestore document that has a key of @type and a string value. We typically have arrays of objects, so something like:

{
  "id": 1234,
  "views": [
    { "@type": "query", operator: "is", value: "some query" }
  ]
}

Expected result

The data gets synced to BigQuery with the right data.

Actual result

An error is thrown with the jsonPayload of:

jsonPayload: {
  @type: "query"
  message: "Data cannot be decoded from JSON."
  operator: "is"
  value: "some query"
}

[ca136]

If you have any ideas on workarounds or temporary patches we could release I'd appreciate it. Would a transform function that renames any @type fields work?

[cabljac]

Hey @ca136 I think a transform function might be a workaround, but not ideal i imagine. I will look into this and see if there's some fix we can make.

Thanks for the including a very detailed description by the way, helps a lot!

[cabljac]

Hi @ca136 I dont think a transform function will work actually because the error is being thrown within onDispatch, and the transform happens afterwards.

You mentioned you updated your extension

We recently updated our extension

Are you able to provide the version number of the extension where you didn't have this issue?

[ca136]

It looks like [email protected] works for sure, we have another sync using this version with the same data. It's likely that versions after this also work, it had been a while since we updated the extentions.

[jauntybrain]

Hi @ca136! Glad to hear this problem doesn't persist in the newer versions of the extension. We will be closing this for now, but feel free to re-open if this occurs again.

[ca136]

I'm not sure I understand what you're saying. The old version of 0.1.20 was working for months and we didn't update until we kept getting pinged by GCP that we needed to upgrade the Node version of these extension functions. We updated to 0.1.43, and this is when we started seeing this bug. Are you saying there's a new version that has a fix?

[jauntybrain]

Hi @ca136,

Apologies for misunderstanding. Our team conducted more tests and uncovered some details of this issue:

• The decode() is indeed causing the problem, and it is triggered by this code when dispatching an enqueued cloud task.
• In this section we serialize data from Firestore onWrite and pass it to the enqueued cloud task.
• The custom payload transform function must be called inside the Cloud Task to allow retry functionality in case of failure.

Given all of the above, this is a complex edge case, and our team is currently discussing possible solutions. We will keep you updated once we implement a fix. Thank you for your understanding.

[shyam-builder]

Hi, Any updates on this? We are still blocked on this, and are now evaluating whether we handwrite our own solution versus continuing to use the extension.

We have a lot of documents with @type field, which are not numbers.

[cabljac]

Hi @shyam-builder

I'll take a look at this escalate to the extensions team again, to see if we can come to a solution that works for everyone. Thanks for your patience!