Matrix generator support for combining generator parameters with matching keys

[jgwest]

This issue and discussion was kicked off from the excellent comment from @winmillwill on the Matrix generator PR.

At present, if the Matrix generator's child generators produce parameters with matching keys, an error will be returned (unless the parameter keys exactly match). For example, @winmillwill noted that both the scmProvider and List generators will try to set a url parameter (ie a parameter named url).

We should definitely allow users to use the Matrix generator with generators that might produce matching parameters, and still allow them to refer to all possible parameters generates by both generators.

My initial thought was something like a paramPrefix field, which would add a user-defined prefix to every key that was generated by that generator:

matrix:
  generators:
    - git:
        repoURL: 'https://github.com/argoproj-labs/applicationset.git'
        revision: HEAD
        directories:
          - path: examples/matrix/cluster-addons/*
      paramPrefix: git. # <--------------------------- This is a new field
    - list:
        elements:
          - cluster: engineering-dev
            url: 'https://1.2.3.4'
            values:
              project: dev
          - cluster: engineering-prod
            url: 'https://2.4.6.8'
            values:
              project: prod
      paramPrefix: list. # <--- New field

So if the list generator above generated (key -> value):

• cluster -> "cluster1"
• url -> "https://kubernetes.default.svc"

The paramPrefix of list. would modify these generated parameters to:

• list.cluster -> "cluster1"
• list.url -> "https://kubernetes.default.svc"

You can see list. is prepended to the key, with list. being the user provided value to prepend. This ensures that (presuming the user provides differing paramPrefixes for each generator) that there can be no overlap, and they can then refer to these parameters independently in the template.

So taken to it's logical conclusion, if the git generator generated:

• path -> (some path)
• path.basename -> (some basename)

It would be modified by the git. paramPrefix:

• git.path -> (some path)
• git.path.basename -> (some basename)

And the Application would be templated with these combined (by the matrix generator) sets of values:

• git.path -> (some path)
• git.path.basename -> (some basename)
• list.cluster -> "cluster1"
• list.url -> "https://kubernetes.default.svc"

(This isn't a perfect example, but hopefully it makes sense :sweat_smile:)

And it might look like this in the template:

template:
    metadata:
      name: '{{git.path.basename}}'
    spec:
      project: some-project
      source:
        repoURL: https://github.com/argoproj-labs/applicationset.git
        targetRevision: HEAD
        path: some-path
      destination:
        server: '{{list.url}}'
        namespace: '{{git.path.basename}}'

Does this make sense? Any obvious flaws with the design, or any scenarios this wouldn't be able to handle? Any tricky implementation details? A better name for paramPrefix? A better alternative idea for how to handle this whole scenario? Feedback welcome!

[winmillwill]

Personally, I'm a little bit perplexed by the non-matrix use case of ApplicationSet and the multiple generators in a generator... those cases seem better served by just making another ApplicationSet. I'd be interested in a successor that just flattened the whole thing so that you have N generators of different types (so generators is an object keyed by type) and they all feed one template, and it generates the matrix of parameters. That means there's no need for a parametersPrefix because you just use the types/keys, so long as there's no compelling use for multiple generators of the same type.

I haven't read up on the original design, and I welcome feedback about the value of a use case I'm not seeing. I also appreciate that this may not be the time to revisit the design completely, but to me it could be on the table for "A better alternative idea for how to handle this whole scenario".

[jgwest]

Thanks for taking the time to write up your thoughts @winmillwill :+1: !

Re: the ability to specify multiple generators at the generators: level: I can see an advantage to this original structure, which does allow you to manage a union of all the values from a single ApplicationSet (rather than needing to keep multiple separate ApplicationSets in sync, reducing 'DRY-ness'). But you are right that this doesn't enable any new use cases (as one could just make a new ApplicationSet and keep the template: portions in sync). I wasn't involved in the original design of the schema so couldn't tell you the specific reasoning here.

Similar to your thinking re: type keys, I had considered just using the name of the generator (eg git. or list.), rather than having a paramPrefix, but assumed that the ability to customize this might be beneficial, plus...

That means there's no need for a parametersPrefix because you just use the types/keys, so long as there's no compelling use for multiple generators of the same type.

Yah, my assumption was that there was a compelling case for multiple generators of the same type (for example, git file x git directory, or list x list, though not with list in its current form). Likewise, if we supported n-dimensional matrices, I can see this being beneficial as well.

[winmillwill]

I want to explicitly say: I think the paramPrefix is a fine, non-invasive change that only makes things better. Anyone who doesn't want to use it doesn't have to.

[LeartBeqiraj]

Is this a supported feature now?