Add support to use template with vault secret multiple times

[ecrupper]

Description

• we have a template which has vault secrets and takes template variables
• we include template once
• But we use it twice in same vela.yml for different purpose controlled by template variable, looks like vela is trying to create vault container multiple times with same name

Value

Any template we are able to use multiple times in vela.yml, but when we add vault in template it fails and we need to do some workaround, which seems to be a drawback

[benzvan]

We're seeing a related issue that is basically the next step on this journey. We have multiple templates that use the same vault secrets. As a result, the vault container is executed multiple times to fetch secrets. We've templated the name of the secret definition to avoid the collision.

I would propose pre-processing the secrets configuration and collapsing each of the secrets using a vault origin with the same authentication so that the vault secret plugin is called once with the complete list of (deduplicated) secrets.

[benzvan]

I think the way I would address this is to add an additional function in types/pipeline/secret.go that could be run after SecretSlice.Purge() like SecretSlice.MergeOrigins()

This function would inspect each Origin in the secret slice and determine if the source information is identical (Environment, Image, Pull, Secrets, and Parameters.username) and merge and dedupe the lists of secrets in Parameters.items into a single Origin. It could either generate a new name from the existing name or create something unique with an additional value to indicate where the secrets were merged from.

If the original issue reported here is a runtime issue and not a configuration-time issue, this would solve both problems.

[GregoryDosh]

The scope of this issue is a little vague since I can think of two different issues it's trying to address. It could be trying to address the scope of name collisions of steps which are used in multiple templates or it could be to consolidate the number of times a plugin is expected to run.

If I focus on the second portion there for a moment, I don't think this is something the Vela server/worker/etc. should be doing. I don't think the pipeline should be enumerated and steps condensed FOR the user since the users should be able to manage their pipelines themselves. That is kind of going against making CI easy to reason with since the behavior changes depending on what sort of plugin or secrets you're pulling and too much magic makes debugging difficult.

Focusing on the specifics of being able to consolidate items being pulled from the Vault plugin, it would make more sense to update the vault plugin to give it the ability to take in parameters and configuration from a file path, and then update the documentation showing a more advanced use case where the plugin is pulled OUT of the top level secrets declaration and is instead part of a steps/stages block as a standalone step of pulling the secrets required.

Instead of...

version: "1"

steps:
  ...

secrets:
  - name: VAULT_USERNAME
    key: SOME_ORG/VAULT_USERNAME
    engine: native
    type: org

  - name: VAULT_PASSWORD
    key: SOME_ORG/VAULT_PASSWORD
    engine: native
    type: org

  - origin:
      name: vault
      image: target/secret-vault:latest
      secrets: [ VAULT_USERNAME, VAULT_PASSWORD ]
      parameters:
        items:
          - source: secret/path/to/loc1
            path: loc1
          - source: secret/path/to/loc2
            path: loc2

You would have something more like...

version: "1"

steps:
  - name: configure items 1
    image: alpine
    commands:
      - echo "source: secret/path/to/loc1,path: loc1" >> /vela/secrets/vault/items

  ...

  - name: configure items 2 (assume this came from a template)
    image: alpine
    commands:
      - echo "source: secret/path/to/loc2,path: loc2" >> /vela/secrets/vault/items

  - name: snag vault secrets
    image: target/secret-vault:latest
    secrets: [ VAULT_USERNAME, VAULT_PASSWORD ]

  ...

secrets:
  - name: VAULT_USERNAME
    key: SOME_ORG/VAULT_USERNAME
    engine: native
    type: org

  - name: VAULT_PASSWORD
    key: SOME_ORG/VAULT_PASSWORD
    engine: native
    type: org

Note that I don't know if those echo statements would work as is, but it gets the point across on how you could update the vault plugin to support this feature without having to touch Vela's non-plugin code.

[benzvan]

I debated if my solution should be a separate issue but it conveniently has the ability to solve this one at the same time, so I went ahead and added it here as a comment. Maybe that's not the right decision.

Since the problem is being created by vela's templating system, it made sense to me that it would be fixed within vela or vela's templating system. It's certainly possible to use the vault secrets plugin or even cURL to get secrets from vault but that basically throws out vela's vault functionality in order to use its template functionality.

The other manual solution we came up with was to create a stand-alone vela template just for vault secrets and add all the secrets all the other templates are using as a list, but that seemed fragile and clunky to do. It may end up being our solution if these two features continue to not work well together.

[GregoryDosh]

The issue with the template system is mostly with name collisions of steps getting duplicated and failing to run because of the re-use of templates. That should absolutely get fixed, but what you're asking for is putting more a coupling between Vela's server/worker codebase and an arbitrary plugin. I think it makes more sense to update that plugin to support more ways of operating than to couple the two even tighter together.