connectors
connectors copied to clipboard
elastic
[CLI] create new API keys for existing connectors
Problem Description
Slack thread: https://elastic.slack.com/archives/C01795T48LQ/p1704810775134149
When doing local development, it's not uncommon for me to configure a connector in my config.yml, but then need to pivot to another branch for some temporary investigation work. When I come back, I've sometimes lost my diff for the config.yml, and that means I need to manually re-configure it with the connector_id, service_type (both still in .elastic-connectors) and api_key.
Proposed Solution
The CLI could do this for me. It could look in .elastic-connectors to see what connectors exist, and then compare with config.yml to see which are missing, and ask the user if they'd like to configure their config.yml to match .elastic-connectors. This would require the CLI to use an elasticsearch user with sufficient permissions to create API keys.
Alternatives
I could keep manually doing this
Additional Context
This could be particularly useful with the Stack Scripts that Mark has been working on, as it would allow you to re-start your stack on a pre-existing volume and fully recover your working state.
In this context API keys already exist for the connectors, right? I assume then that the generating of API keys would also involve invalidating previous API keys.
IDK if there's a good way (today) to trace back to which API key(s) should be invalidated, though. I could also imagine a situation where you have two connectors-python nodes available to run jobs for a given connector. And if you use the CLI to set up both, they'll need to have two different API keys at the same time, so you wouldn't want the second one you set up to invalidate the API key being used by the first one you set up.
We track the API key id on the connector doc under api_key_id so it could be as simple as invalidating what's there, generating a new API key, and saving the new id in its place.
I didn't know we supported multiple API keys for a single connector. Where do we track the excess ids?
We track the API key id on the connector doc under api_key_id
I didn't know that was merged already, but I remember that being part of the secrets storage plan.
I didn't know we supported multiple API keys for a single connector. Where do we track the excess ids?
I honestly hadn't thought about it until this conversation, but there's probably not a logical reason for why we couldn't support multiple api keys for a given connector. Could the secret storage plan be modified to accommodate multiple api keys? Probably ok if not, this is a bit of an edge case - as long as we don't prevent customers from managing and setting their own API keys, I think it's ok if it's just easier for our ui/cli to manage a single key for you.
Sorry, I totally missed your response.
I didn't know that was merged already, but I remember that being part of the secrets storage plan.
It wasn't part of Secrets Storage (that's the api_key_secret_id field). We've always tracked the API key id for connector clients AFAIU, on the field api_key_id. We had this so that Kibana could know what API key to invalidate when rotating an API key or deleting a connector client.
I honestly hadn't thought about it until this conversation, but there's probably not a logical reason for why we couldn't support multiple api keys for a given connector. Could the secret storage plan be modified to accommodate multiple api keys? Probably ok if not, this is a bit of an edge case - as long as we don't prevent customers from managing and setting their own API keys, I think it's ok if it's just easier for our ui/cli to manage a single key for you.
I'm wary of this causing a tangent from the main issue, but just to address your comment; It would be a pretty big task to support multiple API keys, and for possibly minimal gain. Is there a use-case for a user wanting multiple API keys for a single connector?