Additional secret engines for Vault

[jbrockopp]

Description

Original story: https://github.com/go-vela/server/issues/77

I purpose that vela should support more secret engines besides key/value, like gcp, aws, and azure. This change could easily allow administrators to provide a set of credentials to be utilized by tools like terraform without needing to provide long lived credentials.

Some specific design considerations to think of:

• Multiple values are returned by the secret engines mentioned above
  • for example, secret_key, secret_token, lease_duration, etc.
• Should the secret be configured within vault directly or via vela?
  • configuring through vela can be challenging for specific secret engines/configurations due to the number of steps for setup
• Vela specific metadata about these secrets cannot be stored within vault since it doesn't allow additonal metadata like the k/v secret engine does
  • for example, the aws secret engine only allows the following parameters when you add roles https://github.com/hashicorp/vault/blob/961f4468385aa08848baee1b27e64b93904c8bd7/website/source/api/secret/aws/index.html.md#parameters-3, so the metadata about the secret would need to be contained within vela and validated before injection

My initial thought on changes to the pipeline to accomplish this looks like:

version: "1"

steps:
- name: test
  image: alpine
  secrets: [ aws ]
  commands:
    - env | grep AWS_

secrets:
  - name: aws
    key:  aws/sts/foo
    engine: vault
    ttl: 60m

With the step of test returning the output of:

AWS_LEASE_ID=<value>
AWS_LEASE_DURATION=<value>
AWS_LEASE_RENEWABLE=<value>
AWS_ACCESS_KEY=<value>
AWS_SECRET_KEY=<value>
AWS_SECURITY_TOKEN=<value>

The above example showcases that vela will retrieve the credentials through the aws secrets engine and convert all of the returned values to be in the format of <secret name in vela.yml>_<key returned in vault>=<key value>.

My proposed workflow to accomplish the above looks like:

1. Vault administrator enables the new secret engine of AWS within Vault
   • i.e. vault secrets enable aws and vault write aws/config/root access_key=<access key> secret_key=<secret key> region=<region>
2. Vault administrator writes the role(s) they want
   • i.e. vault write aws/roles/<role name> credential_type=<credential type> ...
3. Vela administrator maps the Vault secret with a Vela secret
   • i.e. vela map secret <name of secret within vault> --name <name of secret within vela> --org <org> --repo <repo> ...
4. Vela stores the above metadata about the secret within the Vela database since Vault doesn't allow that type of metadata for non k/v store secret engines
   • some function like https://github.com/go-vela/server/blob/a41ef02c401ba05f8dc229d05de33236f6ceafbc/secret/vault/vault.go#L125 would need to be added to put the data into the vela database
5. User attempts to utilize the newly mapped secret with a pipeline
   • the existing function of https://github.com/go-vela/server/blob/a41ef02c401ba05f8dc229d05de33236f6ceafbc/secret/vault/vault.go#L37 would need to be modified to check the Vela database if the specified secret is not a k/v secret
6. Vela connects to Vault to receive the credentials if the validation to approve injection was successful
7. Vela injects the secrets into the pipeline

Looking for feedback on this before I attempt to write the code to accomplish it.

Value

Definition of Done

Effort (Optional)

1 week

Impacted Personas (Optional)

Vault users