Document usage of Nomad OIDC for external IDPs

[Yethal]

Proposal

Nomad 1.7 added ability to integrate workload identities with external IDPs. It would be super useful if Nomad docs contained setup guides for popular cloud providers such as AWS, Azure etc. For comparison, here's similar guide for setting up Github Actions: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

Use-cases

Using OIDC instead of hardcoded aws credentials in jobs

Attempted Solutions

I really don't want to figure out security settings via trial-and-error.

[tgross]

Hi @Yethal! We've got a tutorial for GCP up already https://developer.hashicorp.com/nomad/tutorials/fed-workload-identity/integration-gcp And this blog post has an example for GitHub Actions: https://www.hashicorp.com/blog/nomad-jwt-auth-with-github-actions

But we definitely want to have tutorials for AWS and Azure as well, they're just going to take a while to get done.

[Yethal]

Hey @tgross. Thanks for the links, unfortunately we use AWS but I'll go through the GCP guide, perhaps the setup is similar enough that this will be sufficient.

[ChefAustin]

For whatever its worth, @Yethal, -- and you're probably already aware of this -- there is an example video here showing a very brief example of using Workload Identity with AWS (which is nice but it does leave much to be desired).

That said, I would also love to see Hashicorp publish some proper documentation on how users can leverage Nomad's Workload Identity with AWS IAM.

(For internal Hashi folks; see: Support Request #150900)

[sofixa]

While we're working on a proper doc, here's an excerpt from my notes describing the configuration flow for AWS:

the Nomad OIDC endpoints (.well-known/openid-configuration and .well-known/jwks.json) need to be publicly reachable over HTTPS on port 443 to be verified. Vault is an exception when [[#JWKS public keys]] are used.

The Nomad servers' configuration file needs to contain the oidc_issuer parameter with the FQDN (including https://) on which the Nomad Servers are available: oidc_issuer = "https://nomad.company.com/"

AWS

OIDC Identity Provider

In AWS, an OIDC Identity Provider needs to be created in IAM. When creating it via Terraform, the API or CLI, the SHA1 thumbprint of the TLS certificate protecting the publicly available FQDN needs to be manually obtained. Via the web console it's automatic.

When using Terraform to configure the Identity Provider, the TLS provider can be used to automatically fetch all relevant thumbprints:

data "tls_certificate" "cert" {
  url = "$NOMAD_ADDR"
}
resource "aws_iam_openid_connect_provider" "nomad" {
  url = data.tls_certificate.cert.url
  thumbprint_list = data.tls_certificate.cert.certificates[*].sha1_fingerprint
}

IAM Roles

Once the Identity Provider is created, AWS IAM roles which allow Nomad workloads to assume them, and give access to AWS resources, can be created. Example AWS IAM Role trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "$OIDC_PROVIDER:aud": "aws"
                }
            }
        }
    ]
}

In this case, the trust policy allows all Nomad workloads with the audience (aud) of aws to assume the role. This ensures that only identities meant for AWS auth are allowed. Any other field from the JWT, such as the nomad_namespace, nomad_task or nomad_job_id can be used to further filter down exactly which workload can assume the IAM role.

Nomad Job file

To provision a JWT identity token that can use the above trust relationship, an identity block is needed on the task level in the Nomad job file:

      identity {
        name = "aws"
        aud = ["aws"] #audience needs to match with the Trust Policy and provider configuration
        file = true #file so that it is stored in a file used by AWS CLI/SDKs
        env = true
        ttl = "1h" #JWT TTL, not related to AWS' 1h default assume role TTL
      }

as well as configuration instructing the AWS CLI or SDK of what role it can assume:

      env {
        AWS_ROLE_ARN = "arn:aws:iam::$AWS_ACCOUNT_ID:role/$ROLE_THAT_CAN_BE_ASSUMED_BY_THIS_IDENTITY"
        AWS_WEB_IDENTITY_TOKEN_FILE = "${NOMAD_SECRETS_DIR}/nomad_aws.jwt" #the name format is nomad_$NAME_OF_IDENTITY.jwt
      }

The AWS SDKs will automatically reauthenticate when their session expires using the latest available JWT, so its TTL is somewhat irrelevant.

[shochdoerfer]

I've documented how to configure Entra ID as OIDC in our company blog some months ago. Just in case anyone needs that https://blog.bitexpert.de/blog/nomad_oidc_entra_id

[SuyashHashiCorp]

Hey @Yethal!! We've published a HashiCorp Knowledge Base Article on this topic. You can find it Integrating AWS IAM with HashiCorp Nomad Workload Identity.

[Yethal]

@SuyashHashiCorp Thank you so much for this

[ChefAustin]

[In case it helps anyone else who comes across this in the future...] If your cluster was initially deployed prior to v1.7.0 release, you might encounter a case whereby your JWT's are being signed with a EdDSA algorithm (which AWS does not support). In this instance, when Nomad presents the JWT, AWS will kick-back and return an InvalidIdentityToken error. To remediate this, you'll need to perform a keyring rotation.

Also, if using the above Terraform code to configure AWS IAM OIDC providers, you might also need to pass in client_id_list to the resource block as the docs have it marked as a required argument).

[Yethal]

Turns out AWS does not support private IDPs (nomad jwks endpoint needs to be reachable from public internet even if the IDP will only be used within internal network) so in the end I won't be able to use this feature (no matter how much I'd want to) but that's AWS fault not Hashicorp's. Perhaps documenting this would be useful so that other people don't waste hours trying to figure out why they're unable to configure this (like I did)

[ChefAustin]

Perhaps documenting this would be useful so that other people don't waste hours trying to figure out why they're unable to configure this

Isn't that what @sofixa commented about above? i.e.

the Nomad OIDC endpoints (.well-known/openid-configuration and .well-known/jwks.json) need to be publicly reachable over HTTPS on port 443 to be verified.

But, fwiw, I went down this road, as well (I think; provided I'm understanding correctly). I was trying to figure out if there's a way to have AWS IAM privately reach out to Nomad's OIDC Provider URL(s) for verification (like some sort of IAM Identity Provider Endpoint that I could deploy within my own VPC to make IAM call the Nomad Provider from inside my own network). Unfortunately, such an "IAM Endpoint" does not exist (not that I had high hopes to begin with; this was merely my own wishful thinking fueled by a hefty dose of Hopeamine™️).

While it certainly won't be feasible for every environment, I think one can publicly expose solely the needed /.well-known/* URLs (while keeping the rest of the Nomad endpoints private) via ALB Listener Rules.

[Yethal]

Isn't that what @sofixa commented about above? i.e.

Looks like I missed that part