Vault integration upgrade based on Workload Identity

[mikenomitch]

trafficstars

Proposal

Once Workload Identity upgrades make it into Nomad, we can redo the Vault integration to use these tokens as the source of auth instead of manually provided Vault tokens.

Using these tokens, Nomad Users would have a one-time set up process to integrate Nomad workloads into Vault.

The general flow for setting up the Vault-Nomad integration would be:

• Set up Vault
  • Create a Policy for Nomad in Vault
  • Enable the JWT Auth Method
  • Configure Vault to use Nomad’s Public keys - either passing in the keys, a JWKS URL, or an OIDC Config URL
  • Create a Vault Role for Nomad
• Set up Nomad
  • Pass a Vault URL into Nomad Server config in a new configuration block (or v2 of the existing vault block). (Note: no token needed)
• Deploy Job
  • Job is configured to use new Vault integration
  • Nomad, recognizing that the new integration is being used, automatically requests a token for this job using the JWT auth method.

This would involve an up front cost to set up roles in Vault, but after that no management of tokens would be needed.

Use Cases & Advantages

This would be advantageous in many ways:

• Nomad Users would not have to manage Vault token issuing, rotation, and revocation for Nomad clients.
• Nomad Users would not have to manage Vault tokens for each Nomad workload.
• Workload identity tokens could be time-bound and automatically rotated
• Workload identity tokens could provide fine-grained access at the task level
• Workload identity tokens could be automatically removed once the task has stopped
• Multiple Vault clusters could be configured to use the same token. This could allow Nomad to talk to multiple clusters at once, handle performance replicas better, and handle disaster recovery better. Failovers could happen without token rotation in Nomad.
• Vault could be more easily deployed as a Nomad job, as client tokens/config would not be needed.

Potential simultaneous improvements

While not directly related, there are a few other Vault improvements that should be considered while we do this upgrade:

• Supporting batch tokens in Vault
• Using Vault secrets in jobspecs, either in artifact stanza or Docker auth (I think this is likely unrelated, but such a popular feature that its worth considering while we implement)