🐛 [firestore-bigquery-export] Cloud function name size gives error on upgrade

[abarone-btf]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: _firestore-bigquery-export
• Extension version: _0.2.5
• Configuration values (redact info where appropriate): N/A

[REQUIRED] Step 3: Describe the problem

I tried to update my extensions from 0.1.X to 0.2.X and it always gives me error for name length. The problem is that I cannot change the name of the extension or the name of the function so the only way I see feasible to upgrade my extensions is to remove it and to create them again (which will cause some problems and inconsistency in my data plus operational overhead).

This is the error when trying to upgrade it:

; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export-player-leaderb-pos/resources/fsexportbigquery: {"ResourceType":"gcp-types/cloudfunctions-v2beta:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Invalid Request. Violations: {Violation in Function.name=Invalid function_id. Must start with a letter, can only contain [-_a-zA-Z0-9] and be between 4-63 characters.}","status":"INVALID_ARGUMENT","statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v2beta/projects/MY_PROJECT_ID/locations/us-central1/functions","httpMethod":"POST"}}

I think prior to 0.2 the cloud functions name was cut when surpassing the length (because it never gives my trouble, and some functions in fact get cut, like "ext-firestore-bigquery-export-player-communities-fsexportbigque"), something of that sort can be made here?, to avoid this type of error which are very annoying in terms of resolution from my end :(. Or is it a better way?

EDIT: Looks similar to 2409

Thanks for your help

Steps to reproduce:

Just upgrade the extension using the console, keeping all my previous values and default settings

Expected result

To upgrade the extension without problems, and if the function name is greater in len to cut them down, or to allow resolution when updating it (to avoid deleting and re creating)

Actual result

Explained above, but here is the error message:

; RESOURCE_ERROR at /deployments/firebase-ext-firestore-bigquery-export-player-leaderb-pos/resources/fsexportbigquery: {"ResourceType":"gcp-types/cloudfunctions-v2beta:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"Invalid Request. Violations: {Violation in Function.name=Invalid function_id. Must start with a letter, can only contain [-_a-zA-Z0-9] and be between 4-63 characters.}","status":"INVALID_ARGUMENT","statusMessage":"Bad Request","requestPath":"https://cloudfunctions.googleapis.com/v2beta/projects/MY_PROJECT_ID/locations/us-central1/functions","httpMethod":"POST"}}

[cabljac]

Hi, i have raised this with the extensions platform team to investigate, and will provide updates when available!

[abarone-btf]

Hi @cabljac any news about this?? 🙏

[abarone-btf]

Sorry to bother, just to have another person aware of this issue 🙏 . Do you know if there's any news about this? @HassanBahati

[cabljac]

Hi all, the team is meeting later today so i'll be able to provide any updates.

[abarone-btf]

Any news @cabljac ? 🙏

[cabljac]

Ah apologies yes. I've raised it with the team.

Im not 100% on things as i dont have direct access to the backend code, but I believe the extensions platform is responsible for truncating function names longer than 63 characters. I've seen a snippet of code that seems to be doing this.

My theory is that there's a bug where v2 functions aren't hitting this logic for some reason. I can't see a problem in the code I've seen though. Ive raised this with the extensions backend team, and will relay any updates.

The only workaround for now is to use a shorter extension instance ID

[abarone-btf]

Thank you very much @cabljac !. How can I use that workaround?, when upgrading I can't change neither the instance name nor the function name :(

[cabljac]

You might be able to export the manifest first and then change it

We have a similar issue here where this process is explained:

https://github.com/firebase/extensions/issues/2457

We will try to repro and ensure this workaround is actually a workaround. Thanks for your patience!

[abarone-btf]

Hi @cabljac !, (sorry to bother again), did you have the time to test the workaround to see if indeed does work?

Thank you!

[cabljac]

Hey! Sorry not yet, some high priority stuff got in the way, i'll try and get this prioritised higher

[abarone-btf]

Hi @cabljac ! (or @CorieW , @HassanBahati ), (sorry to bother again), did you have any news about this problem :( ?

Thank you!

[abarone-btf]

Any news about this issue? @cabljac 🙏

[abarone-btf]

Hi @CorieW @HassanBahati, could you please help us with this issue?
In October, the generated functions I have will be deleted due to runtime deprecation, and I don’t have a way to upgrade them gracefully because of this limitation.

🙏 @cabljac

[cabljac]

Hey @abarone-btf sorry for not responding. I'll pursue this with the firebase team and see what we can do. Thanks for your patience!

[cabljac]

@abarone-btf So if you do firebase ext:export in the firebase CLI, you should be able to export your extension configuration to a local .env and firebase.json (from me testing, it will present changes to review, and then you hit Enter to confirm)

You should then be able to change the name of the extension in that firebase.json (and also the name of the .env file!) to something shorter, and firebase deploy --only=extensions should uninstall your extension and install a version of the extension with a shorter instance id (which becomes part of the deployed functions)

From there you should be able to update the extension without error.

This is a workaround because it requires you to change the extension instance id and reinstall, so it's not perfect. I'll provide updates from the team as soon as i get them

[abarone-btf]

Hi @cabljac, thank you very much for your help. We looked into that method, but we’re concerned that during the uninstall/reinstall process some data might be lost (and never reach BigQuery). Is that something we should be worried about, or is it fully managed by the deployment process internally?

Thanks again!

[cabljac]

Hi @abarone-btf - this is a valid concern I think.

If you install the new instance first and then uninstall the old instance, it should avoid data loss in the crossover, but you'll likely get some duplicate events through. I think the "latest view" should deduplicate the data, so it depends how you're using the data downstream.

I think deleting this duplicate data may be tricky, since the table is being streamed to. I will investigate further and get back to you.

[abarone-btf]

Hi! @cabljac , thank you for your respond!. But the workaround mention that the firebase deploy should uninstall your extension and install a version of the extension with a shorter instance id, that would be automatically?, or I need to install a new one and then uninstall the older one manually?

[cabljac]

To be sure, I would manually install both, i'm not sure order is guaranteed at all. If you try to do both at once it may delete the old one before the new one is installed.