Add support for passing identity information from Nomad to Vault

[ddaws]

Proposal

It would be useful to be able to pass identity information from Nomad to Vault to support creating template policies for services. Currently the Nomad to Vault integration does not pass any identity information, so each service ends up having it's own policy.

For example, given two Nomad jobs like

# job-abc.nomad
job "abc" {
  # ...
}

job "xyz" {
  # ...
}

I need to create two policies in Vault to provide access to different paths in the KV store

# service-abc.hcl
path "secrets/data/services/abc" {
  capabilities = [ "read" ]
}

# service-xyz.hcl
path "secrets/data/services/xyz" {
  capabilities = [ "read" ]
}

It would be great if this could be expressed as a single dynamic policy like

# service.hcl
path "secrets/data/services/{{ identity.entity.name }}" {
  capabilities = [ "read" ]
}

Use-cases

The primary use case is providing consistent access to Vault for services. In my toy example I talked about two services but in practices teams are often running 10s or 100s of jobs in Nomad and most jobs need roughly the same access to their own secrets, with some jobs needing extra access that could be provided through additional policies.

Attempted Solutions

I am currently solving this using Terraform to manage policies in Vault and the templatefile function in Terraform, but this still requires a change to my Terraform code for every additional service (needs to be added to the list of services and therefor needs and MR and review). Templating policies through Terraform still results in a large number of static policies being created in Vault which makes the Vault UI and CLI less usable.

[ddaws]

I would be surprised if this hasn't already been requested so maybe this issue is a duplicate. I was asked to open a feature request here so just making sure it gets tracked: https://discuss.hashicorp.com/t/how-can-i-create-a-vault-template-policy-referencing-nomad-job-metadata/41879/3

[jrasell]

Hi @ddaws and thanks for raising this. I am cross linking to #14764 which is related to ACLs in a slightly different manner and will mark this appropriately on our backlog board.

[vftaylor]

+1 for this request. This feature is a must IMO. Currently, we have to create a distinct policy for each Nomad job and it needs to be manually attached to the job spec. This means that we can't have the end user write their own job spec for security reasons, because they would be able to specify arbitrary policies to attach to their jobs. Dynamic policies would go half the way to solving this problem.

[ddaws]

@vftaylor you should look at implementing a Sentinel policy that limits the policies that are associated to a job. For example, you could enforce one policy per job, and that the policy name matches the job name. You still need to pay attention to what is in the policy but you could allow users to write their own job specs and feel confident they can't request arbitrary Vault policies.

[vftaylor]

@vftaylor you should look at implementing a Sentinel policy that limits the policies that are associated to a job. For example, you could enforce one policy per job, and that the policy name matches the job name. You still need to pay attention to what is in the policy but you could allow users to write their own job specs and feel confident they can't request arbitrary Vault policies.

Thanks, that is a good suggestion @ddaws; I never considered it. Ideally though, this sort of thing would receive first class support from Nomad/Vault to obviate the need for an additional policy enforcement engine.

[ddaws]

@vftaylor how would that work? Or what might it look like in HCL terms?

Nomad could supply identity info to Vault and that would allow us to create a generic policy for services but that wouldn't really prevent someone writing a Nomad job spec from requesting additional policies. For example, given the generic policy

# service.hcl
path "secrets/data/services/{{ identity.entity.name }}" {
  capabilities = [ "read" ]
}

My Nomad job specs could be simplified to all request the same policy like

job "docs" {
  group "example" {
    task "server" {
      vault {
        policies = ["service"]
      }
    }
  }
}

But there is nothing stopping the author from including an additional policy like

vault {
  policies = ["service", "extra-secret-stuff"]
}

Maybe I'm misunderstanding though. I'm interested to hear what you are thinking!

[vftaylor]

@ddaws perhaps at the Nomad server config level, you'd be able to specify that all Nomad jobs get that generic Vault policy assigned by default. And then you could have a flag that, if set, restricts all jobs to that single policy regardless of what the job spec says. So a job spec need not even have a vault stanza at all, which cuts down on repetitive code.

Jobs would only be able to override this default Vault policy if they were able to prove that they were "privileged" in some way. Maybe by providing some authentication token. I've not given much thought to this last part though.

[nvx]

Personally the usecase I think this would be most helpful for is just to save on amount of config needed. Assuming your Nomad operators are trusted, the code running in your jobs may not necessarily be (or may have bugs that allow code execution inside the eg docker container that they run in) - so limiting the blast radius of a specific job is useful, so being able to restrict jobs to only be able to read from "secrets/data/nomad/" is useful for preventing jobs from reading other jobs secrets in Vault - but today requires a separate policy for every nomad job to achieve.

Similar applies with dynamic secrets engines (databases, PKI, etc) where you might want to by convention restrict jobs to only being able to access eg database roles with the same name as the job name, or only being able to create TLS certificates where the common name is .service.consul. for example.

[ljb2of3]

I've ended up here after trying to figure out how to configure nomad and vault to do exactly this.

I'm in agreement with @nvx. My ops team is very small, and we're all trusted with access to everything in vault since we're the ones entering the secrets in the first place, but we'd like to stick to the principle of least privilege when writing our jobs so that a compromised process can't access more than it needs.

We can also include an internal security review step to our CI/CD pipeline before jobs are allowed to be submitted so the policy set is double-checked to ensure someone isn't writing a job requesting access to policies they don't need.

[mikenomitch]

Hopefully we can do this as part of the Vault Integration Refresh using workload identity - slated tentatively for 1.7

(Also noting the Vault Templated Policies docs for future reference.)

[mikenomitch]

Code for this is merged and coming in 1.7 GA