🐛 [firestore-bigquery-export] dataloss when reconfiguring extension

[cabljac]

Opening this issue to track something raised in the comment section of another issue.

The issue seems to be with the design of the extension/the extensions platform for reconfiguration.

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

See the original comment here

https://github.com/firebase/extensions/issues/2133#issuecomment-2250860395

Expected result

The cloud task queue not to have been wiped

Actual result

it gets wiped

[cabljac]

Im not sure how we'd resolve this situation within the extension to be honest. We would have to somehow preserve the queued cloud tasks across reconfiguration. Best advice might be not to pause the queue, to minimise data loss.

[cabljac]

ah so there's a workaround I thought of, but it would involve a second extension instance

1. install new instance of extension to write to another table
2. reconfigure original extension instance
3. once the extension is reconfigured and streaming events, uninstall the second instance
4. run a suitable BQ job to merge tables

[Nushio]

Pausing was only meant as an example to quickly replicate. In my original case, I had around 200,000 back-logged documents (because i set the number of documents to sync to 10) and they vanished when I raised it to 100. I didn't pause it but didn't expect it to be gone either.

The second extension would probably be a good alternative, alongside placing a warning/disclaimer somewhere about this (potential) data loss.

Thanks!