ExternalSecretSet: Generate mutliple ExternalSecret from a path with wildcard

[speedfl]

Is your feature request related to a problem? Please describe.

In some cases it could be interesting to generate multiple ExternalSecret based on a path.

Example:

I have several secrets with same format (ie: argocd repositories/db connection strin) under a path.

Example:

/teams/teamone/argocd/repositories/repo1
/teams/teamone/argocd/repositories/repo2
/teams/teamone/argocd/repositories/repo3
/teams/teamone/argocd/repositories/repo4

The format is always:

{
   "repoURL": "http://github.com/test/test.git",
   "project": "teamone",
   "username": "argocd",
   "password": "123456"
}

Describe the solution you'd like

I want an auto discovery based on /teams/teamone/argocd/repositories/* so that each time a new repository is added in /teams/teamone/argocd/repositories/ an ExternalSecret is created

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecretSet
metadata:
  name: "hello-world"
spec:
  syncPolicy:
    childPolicy: "create-only" # create-only / create-delete / create-update / sync
  refreshTime: "1m"
  generators:
      - path:  `/teams/teamone/argocd/repositories/*` # This use the same provider as for the ExternalSecret (example: Vault)
  template: # This is an ExternalSecretTemplate
    metadata:
      labels:
        key: value
      annotation: 
        key: value
      name: "teamone-{ .key }-secret"
    spec:
      secretStoreRef:
        name: secret-store-name
        kind: SecretStore

      refreshInterval: "1h"

A ownerReference must be added to the generated ExternalSecret

Describe alternatives you've considered N/A

Additional context N/A

[gusfcarvalho]

Hi @speedfl .

Edit: just properly read it.

I see many issues with this proposal as is. Where would this path come from? How would you prevent circular dependencies between external secrets?

IMO, this is a templating problem which would be a bit far off for us to solve within ESO itself. IMO, you would be better off with any existing templating tools out there like helm.

[speedfl]

Hello Sorry if it was not clear. The goal is not to look in git but all keys in a path in the targeted provider Example all keys in vault under /teams/teamone/argocd/repositories/*

I updated the manifest for the example

[moolen]

Hey, thanks for this proposal! I see the auto-discovery use-case, other users requested that as well.

If I understand correctly, the idea is to have one Kubernetes secret per provider secret under a given path /teams/teamone/argocd/repositories/*.

By design, Kind=ExternalSecret manages only a single secret. Looking at the code, changing that behaviour is hard, because we ~~need~~ want to support creationPolicy/ mergePolicy / deletionPolicy and all that stuff. My gut feeling tells me that the autodiscovery feature can be built into a ExternalSecret (which doesn't mean it should) , and it's likely less overhead compared to creating a new CRD with a controller, all the documentation and productionizing it.

The biggest problem would probably be how to store the state (of which secrets are provisioned) and how to keep a sane ExternalSecret codebase :thinking:

I don't want to "just" generate an ExternalSecret, there are other tools that help with that simple task. I don't think we should reinvent the wheel unless we have a good reason to.

This Kyverno policy looks promising, that would solve the problem for your use-case. Alternatively, you could achieve that with a couple of bash scripts and do this async.

These are just my initial thoughts on this..

[Aransh]

Hey there, joining here from my duplicate issue - https://github.com/external-secrets/external-secrets/issues/2987#issuecomment-1873351835

I have the exact same use case (you can see my closed ticket for more details), where the current approach is much too static in case I want to pull keys/values from more secrets (or less, in case a secret exists in one path and not the other), adding configuration complexity from the user side.

@moolen I will mention, your response mentions the complexity of one "externalSecret" pulling from several secrets, but this is already a supported use-case in external-secrets, as we can add multiple "dataFrom" in a single "ExternalSecret"... Only issue I would see here in comparison, is how to prioritize which one overwrites which in case of key duplication, in which case I would say probably best to just keep as "prioritization is by alphabetical order" or something, leaving it up to the user to name the secrets appropriately (possibly "0-overwrite" and "1-default" for example.

Hope I made myself clear, and hope you will consider implementing this, as the flexibility it adds is very valuable for agility.

[Aransh]

Just took a second look, and I seem to have misunderstood this issue. You are requesting creating multiple secrets from all secrets in a directory I am requesting creating a single, merged secret from all secrets in a directory.

I see value in both approaches, and they could both be solutions to similar issues, but I think mine seems to me aligns more with the existing external-secrets feature set. Also It’ll enable simple secret mounting (as you can’t mount a secret using a wildcard).

Anyway, don’t mind me, just showing my support for another ticket with similar nature 🙂

[NetanelK]

We have a similar use case. We use ESO to pull cluster info into Argo CD, and we’d also like to have a single central manifest instead of hundreds of ExternalSecret manifests rendered by Helm from a static list. That approach isn’t dynamic enough for us either.

I’d love to help out with this one!

[gusfcarvalho]

We have a similar use case. We use ESO to pull cluster info into Argo CD, and we’d also like to have a single central manifest instead of hundreds of ExternalSecret manifests rendered by Helm from a static list. That approach isn’t dynamic enough for us either.

I’d love to help out with this one!

Hi @NetanelK . The first step is to provide a design proposal. Your focus there should be both the CRD modifications you intend to do, plus the code changes to make this viable. You should also cover how you’ll address state tracking on ExternalSecrets.

[NetanelK]

@gusfcarvalho I’d like your input on the direction of this proposal.

I have two main design paths:

1. Create a new CRD - ExternalSecretSet This would act as a generator that discovers secrets under a given path and creates multiple ExternalSecret objects. It keeps the existing CRD simple, but the new one doesn’t really add much value besides creating multiple secrets at once. It also adds another resource type for users to learn and manage.

2. Extend the existing ExternalSecret CRD Here we’d allow wildcards in remoteRef.key, so a single ExternalSecret can represent a one-to-many relationship. The controller would handle discovery, reconciliation, and state tracking (via something like a status.generatedSecrets field). It means slightly more logic in the controller, but a cleaner experience for users since everything stays under the familiar ExternalSecret resource.

Right now I’m leaning toward extending ExternalSecret, since it feels more natural and avoids introducing an extra CRD that doesn’t bring much functional gain.

What do you think?

[gusfcarvalho]

As I said above, I think this deserves a proper design document first - they’re available on a design folder, submit a draft PR there for all maintainers to look

ExternalSecret object by design was meant to not allow one-to-many relationships. ExternalSecretSet seems more of a reasonable proposal to me, though I’m still not convinced this is a problem we should solve at this level, but I’m happy to read the proposal and see if it changes my mind 🙂