Use same secret key to manage different functions

[elhe26]

Description

As the title implies, please allow using the same key for multiple functions. It's expensive to create the same key over and over again just to be used by 1 function.

Steps to reproduce

NA

Expected Behavior

One secret key for multiple functions.

Actual Behavior

The same key is created and used per function.

Metadata

Typesense Version: 0.27.1

OS: MacOS

[jasonbosco]

Could you elaborate on this? I'm not sure I understand why you can't use the same API key for multiple instances of the extension?

[elhe26]

Of course @jasonbosco . When using the extension, you create a secret each time you define the function. If you have 5 collections, you need 5 secrets, one per collection, using the same Typesense API key. The configuration must have a way to use the previously created secret from the first function on others.

[jasonbosco]

Ah I see what you mean now. I wonder if this is a Firebase UI limitation that doesn't let you pick an existing secret? You might want to try using the Firebase CLI to install the extension and then it might allow you to reference an existing secret by its resource ID? (I'm not too familiar with this though - might be a better question for Firebase support).

[elhe26]

Migrating to v2 allows you to use defineSecret by creating a secret key first (ie. "TYPESENSE_API_KEY" on secret manager) and then using that variable with that function (defineSecret). I don't remember being able to do that on v1.

[jasonbosco]

May I know v2 of what package you're referring to?

[stevebrowndotco]

Im finding this to be a limitation when having multiple extensions for different tyepsense collections when adding the local config and deploying via the CLI. This is the error I get in the cli. Therefore I can confirm a limitation with GCP?

Error: firestore-typesense-search-omitted: Found 'projects/OMITTED/secrets/firestore-typesense-search-TYPESENSE_API_KEY/versions/latest' for secret param TYPESENSE_API_KEY. projects/OMITTED/secrets/firestore-typesense-search-TYPESENSE_API_KEY is managed by a different extension instance (true), so reusing it here can lead to unexpected behavior. Please choose a different name for this secret, and rerun this command.

[elhe26]

May I know v2 of what package you're referring to?

I'm referring to Cloud Functions V2.

Dependencies

...

  "dependencies": {
    ...
    "express": "^4.21.2",
    "firebase": "^11.0.2",
    "firebase-admin": "^13.0.1",
    "firebase-functions": "^6.1.1",
    "typesense": "^1.8.2"
  },
  ...

Firebase Cli

firebase-tools: 13.20.2.

[elhe26]

Interesting @stevebrowndotco . Could you please share the prompt you used to configure the ext? I see that this ext adds the following parameters when creating the secret key: Secret Manager Admin and Secret Manager Secret Accessor. I'm using other keys in different settings without any issues.

[stevebrowndotco]

@elhe26

1. I am running firebase ext:install typesense/firestore-typesense-search
2. It then creates a local .env file with the following entry:

TYPESENSE_API_KEY=projects/omitted/secrets/ext-firestore-typesense-search-TYPESENSE_API_KEY/versions/1

3. HOWEVER, since I already use the above key , i get the following error:

Error: firestore-typesense-search: Found 'projects/omitted/secrets/ext-firestore-typesense-search-TYPESENSE_API_KEY/versions/1' for secret param TYPESENSE_API_KEY, but this instance was previously using a different secret projects/omitted/secrets/firestore-typesense-search-TYPESENSE_API_KEY.
Changing secrets is not supported. If you want to change the value of this secret, use a new version of projects/omitted/secrets/firestore-typesense-search-TYPESENSE_API_KEY.You can create a new version at https://console.cloud.google.com/security/secret-manager?project=omitted

4. In my case, make sure the instance name is different to fix the above.

5. I came to this issue only because I created the extension before google secrets was supported in this library. So for me, the solution was to manually go into the "secrets manager" in GCP, and create an entirely new secret, and modify this line in firestore-typesense-search.env file:

TYPESENSE_API_KEY=projects/omitted/secrets/ext-firestore-typesense-search-TYPESENSE_API_KEY/versions/1

To (for example)

TYPESENSE_API_KEY=projects/omitted/secrets/ext-firestore-typesense-search-ANOTHER-TYPESENSE_API_KEY/versions/1

Also trying multiple versions in the same key will not work either.

[tharropoulos]

@elhe26 We've published the v2.0.0-rc.3 release that includes V2 functions, could you give it a try?

[elhe26]

@elhe26 We've published the v2.0.0-rc.3 release that includes V2 functions, could you give it a try?

Will do. Thanks!

[elhe26]

@tharropoulos , is there a way to add an extra button to update/use a different key?

[tharropoulos]

@tharropoulos , is there a way to add an extra button to update/use a different key?

You can update the configuration by hitting the "manage" button in your extensions panel. That includes the API key

[elhe26]

@tharropoulos , is there a way to add an extra button to update/use a different key?

You can update the configuration by hitting the "manage" button in your extensions panel. That includes the API key

@tharropoulos , the main issue is that each extension creates a new secret key. Is there a way to use the newly created key from the first extension on a second ext, third ext, etc.? We can remove the secret key from the second ext because there's no way to configure the secret anywhere.

Edit:

If we go to google cloud console, then secret manager, then remove the secret key associated with the second ext, this could cause an error since the second ext. won't be able to find the secret key since it doesn't exist anymore.

[tharropoulos]

Is there a way to use the newly created key from the first extension on a second ext, third ext, etc.?

Keys themselves are just Typesense API keys. You can use a single one everywhere, without needing to regenerate each one for each extension installation. Does the UI not let you paste in the form element?

[elhe26]

Is there a way to use the newly created key from the first extension on a second ext, third ext, etc.?

Keys themselves are just Typesense API keys. You can use a single one everywhere, without needing to regenerate each one for each extension installation. Does the UI not let you paste in the form element?

Every time we add an extension, a secret key is created. I'm using the same Typesense API key for each extension.

When I remove the key from the second extension from Google Cloud console and go to the extension UI and click on "reconfigure extension", I get the following:

When I try to change the path to use the key from the previously created key (first ext) and click create secret, this procedure will ignore the path and create a new key again.

[tharropoulos]

The Key should be a Typesense API key, not a path. You can see your API keys by calling out to the /keys resource in the Typesense API:

https://typesense.org/docs/28.0/api/api-keys.html

[webpepper]

I also find it limiting that it's not possible to store the Typesense key as a secret only once (even if it had to be done manually) and then just reuse this secret in as many extension installations as I want.

It's clear the extension is currently tailored to be used with a single collection. It works but it's not very practical since often there are multiple collections that need to be indexed. E.g., if I want to index collections like "movies", "actors", "genres" I have to install 3 instances of the extension, and each has to be configured from scratch. And that includes the Typesense key, which gets saved as a brand new secret each time.

[jasonbosco]

We're planning to tackle the challenges of managing multiple instances of the extension in this thread: https://github.com/typesense/firestore-typesense-search/issues/121