firebase
🐛 [Stream Firestore to BigQuery] Limiting or Controlling the number of resources used for extension.
[READ] Step 1: Are you in the right place?
Issues filed here should be about a feature request for a specific extension in this repository. To file a feature request that affects multiple extensions or the Firebase Extensions platform, please reach out to Firebase support directly.
[REQUIRED] Step 2: Extension name
This feature request is for extension: firebase/firestore-bigquery-export
What feature would you like to see?
I would like you to consider several options for the development of the extension system:
- Installation of an extension for several collections.
- Limiting or Controlling the number of resources used for extension.
Why?
Today, due to an error in firebase-tools, all my extensions were erased and I had to reinstall them several times.
I have 8 collections that I export to BigQuery, that is I need to install firebase/firestore-bigquery-export 8 times.
Each instance creates:
- 5 Firebase functions
- 4 queues (when extension deleted, it turned out that they are not deleted and you need to delete them manually)
Thus, after installing 8 times, I got 40 Firebase functions and 32 queues.
Before that, I used several versions of this extension. The first version assumed only 1 function. That is, I would have 8 functions, not 40. And not a single queue. But as a software architect, I understand that this option has poor scaling. After that, you implemented a more fault-tolerant and scalable architecture using 1 queue and two functions (producer and consumer). I used this option before the error that caused me to reinstall all extensions, i.e. I had 8 queues and 16 functions. It was ok, but it could also have been optimized. For example, you could use 1 queue and one consumer for all extensions.
At the same time, the functions have not been optimized in terms of resources consumed. They work fine on 128MB of memory, but the default is 256MB. At the same time, when the extension is reconfigured, the setting drops back to 256MB. At the same time, only the producer is limited to 3000 instances, and 4 more functions are not limited by the number of copies. And this is for each of the 8 extension instances.
At the same time, when reinstalling and reconfiguring, re-synchronization is now started, which also consumes resources.
The point of the extension is that it performs some simple functionality. And this extension really does not do anything complicated except for streaming insertion into the dataset table. But you use so many resources for this, that rather than spending time monitoring and limiting them, it is easier to abandon its use and write your own solution (it will hardly take more than a couple of days). Are you sure you turned that way?
How would you use it?
Control over the resources used.
These are valuable and valid points about resource optimization. I'll bring these concerns to the team.
While refactoring work can be challenging to prioritise, your detailed examples help make a stronger case for improvements, and are appreciated!
Thank you!
On Tue, Mar 4, 2025 at 13:35 Jacob Cable @.***> wrote:
These are valuable and valid points about resource optimization. I'll bring these concerns to the team.
While refactoring work can be challenging to prioritise, your detailed examples helps make a stronger case for improvements, and are appreciated!
— Reply to this email directly, view it on GitHub https://github.com/firebase/extensions/issues/2164#issuecomment-2696819512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXASJWNILWSQWCRIK3LTED2SVXXDAVCNFSM6AAAAABM6DYB5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWHAYTSNJRGI . You are receiving this because you authored the thread.Message ID: @.***> [image: cabljac]cabljac left a comment (firebase/extensions#2164) https://github.com/firebase/extensions/issues/2164#issuecomment-2696819512
These are valuable and valid points about resource optimization. I'll bring these concerns to the team.
While refactoring work can be challenging to prioritise, your detailed examples helps make a stronger case for improvements, and are appreciated!
— Reply to this email directly, view it on GitHub https://github.com/firebase/extensions/issues/2164#issuecomment-2696819512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXASJWNILWSQWCRIK3LTED2SVXXDAVCNFSM6AAAAABM6DYB5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJWHAYTSNJRGI . You are receiving this because you authored the thread.Message ID: @.***>
I think there are two issues at play:
-
this specific extension is not very optimised for resources used, and that leads to a lot of cleanup, especially with multiple versions of the extension
-
there's no great way of specifying a trigger for firebase functions for a finite list of collections for example. We only have collection groups.