Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace

[mkilchhofer]

Is your feature request related to a problem? Please describe. In our current setup, our K8s platform customers use https://github.com/postfinance/vault-kubernetes to sync Secrets from Vault OSS into their desired Kubernetes namespace.

Our Vault is fully configured via Terraform. The k8s integration is done via Kubernetes Auth method on namespace basis (a Vault role per K8s namespace). In detail, we have this structure:

module "my_k8s_cluster" {
  source             = "./k8s-clusters"
  cluster_name       = "my-k8s-cluster"
  kubernetes_host    = "https://<api-server-url>"
  kubernetes_ca_cert = <<EOF
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF

  disable_iss_validation = true
  reviewer_jwt           = var.my_k8s_cluster-jwt
  namespaces = {
    # Customer namespaces
    "app-a-namespace" = {
      secret_engine = "solution/my-solution-a/blabla"
      secret_path   = "general/my-app-a"
    }
    "my-app-b-namespace" = {
      secret_engine = "solution/my-solution-b/blabla"
      secret_path   = "github-actions/*"
    }
    "my-app-c-namespace" = {
      secret_engine = "solution/my-solution-c/blabla"
      secret_path   = "test/*"
    }

    # And also some platform namespaces
    "test-vault-sync" = {
      secret_engine = "realm/k8s/kv"
      secret_path   = "testing/test-vault-sync/*"
    }
    "monitoring" = {
      secret_engine = "realm/k8s/kv"
      secret_path   = "alertmanager/*"
    }
  }
}

This module then installs multiple config resources inside Vault:

########################
# Configure Vault to talk to Kubernetes
########################
resource "vault_auth_backend" "kubernetes" {
  type        = "kubernetes"
  path        = "k8s-${var.cluster_name}"
  description = "kubernetes auth for cluster ${var.cluster_name}"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend            = vault_auth_backend.kubernetes.path
  kubernetes_host    = var.kubernetes_host
  kubernetes_ca_cert = var.kubernetes_ca_cert
  token_reviewer_jwt = var.reviewer_jwt

  disable_iss_validation = var.disable_iss_validation
}

########################
# K8s Namespace to Policy mapping
########################
resource "vault_kubernetes_auth_backend_role" "vault_sync" {
  for_each = var.namespaces

  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "vault-sync-${each.key}"                     # <-- see here
  bound_service_account_names      = ["vault-sync*"]
  bound_service_account_namespaces = [each.key]                                   # <-- here
  token_policies                   = ["k8s-${var.cluster_name}-${each.key}-read"] # <-- and here
}

########################
# Read policies per Namespace
########################
data "vault_policy_document" "k8s-policies" {
  for_each = var.namespaces

  rule {
    path         = "${each.value.secret_engine}/data/${each.value.secret_path}"
    capabilities = ["read"]
  }
  rule {
    path         = "${each.value.secret_engine}/metadata/${each.value.secret_path}"
    capabilities = ["list"]
  }
  rule {
    path         = "sys/mounts"
    capabilities = ["read"]
  }
}

resource "vault_policy" "k8s-policies" {
  for_each = var.namespaces

  name   = "k8s-${var.cluster_name}-${each.key}-read"
  policy = data.vault_policy_document.k8s-policies[each.key].hcl
}

So in practice we have:

• 1 vault_auth_backend per cluster
• 1 vault_kubernetes_auth_backend_config per cluster
• N vault_policy per cluster (1 per namespace)
• N vault_kubernetes_auth_backend_role per cluster (1 per namespace)

Describe the solution you'd like It would be nice to share both

• the vault default Connection
• the vault default Auth (Kubernetes Auth)

with all customer namespaces, so they only need to define the VaultStaticSecret etc. :

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: default
  namespace: vault-secrets-operator                  # <-- sits in the VSO namespace
spec:
  method: kubernetes # same as tool from PF
  mount: {{ printf "k8s-%s" .Values.clusterName }}
  kubernetes:
    role: vault-sync-vso
    serviceAccount: vault-sync-vso
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: default
  namespace: vault-secrets-operator                  # <-- sits in the VSO namespace
spec:
  address: https://vault.example.com
  caCertSecretRef: vault-ca

Describe alternatives you've considered Only sharing the kind: VaultConnection with all namespaces and let our platform customers define the kind: VaultAuth in each namespace.

Additional context

I did a short PoC without spending too much time on the golang code which worked:

diff --git a/internal/credentials/vault/kubernetes.go b/internal/credentials/vault/kubernetes.go
index 830c1f4..0ed9af2 100644
--- a/internal/credentials/vault/kubernetes.go
+++ b/internal/credentials/vault/kubernetes.go
@@ -5,6 +5,7 @@ package vault

 import (
        "context"
+       "fmt"

        corev1 "k8s.io/api/core/v1"
        "k8s.io/apimachinery/pkg/types"
@@ -82,9 +83,11 @@ func (l *KubernetesCredentialProvider) GetCreds(ctx context.Context, client ctrl
                return nil, err
        }

+       role := fmt.Sprintf("%s-%s", l.authObj.Spec.Kubernetes.Role, l.providerNamespace)
+
        // credentials needed for Kubernetes auth
        return map[string]interface{}{
-               "role": l.authObj.Spec.Kubernetes.Role,
+               "role": role,
                "jwt":  tr.Status.Token,
        }, nil
 }

Is this a valable approach? If so, I could invest some more time to make it configurable.

Maybe somehow related:

• #274

[mkilchhofer]

ping @benashz

[mkilchhofer]

@benashz @tvoran any feedback on this?

[Fantaztig]

Imho this doesn't make much sense without specifying a ServiceAccount for each namespace to use for authentication as well. Otherwise your VSO SA basically has full read access to all secrets because it can assume all roles. I understand the role a namespace is using is supposed to be fixed in that scenario but still it feels a bit weird.

What keeps you from generating a VaultAuth for each namespace? Don't you already have a template for common namespace resources where this would fit neatly e.g. for RoleBindings etc.? We used to include this so a platform customer could simply tick "Vault" when ordering a namespace and everything came already generated for them (both on k8s and vault side).

[mkilchhofer]

Imho this doesn't make much sense without specifying a ServiceAccount for each namespace to use for authentication as well.

Otherwise your VSO SA basically has full read access to all secrets because it can assume all roles. I understand the role a namespace is using is supposed to be fixed in that scenario but still it feels a bit weird.

That is true in my opinion if you use a default VaultAuth:

https://github.com/hashicorp/vault-secrets-operator/blob/c5fa7cf740f9a3c9721946a0aa5c9876722cb059/chart/values.yaml#L591

And you are right this does not make sense and IMHO this is a security flaw. What I want is that we can configure a static name of a ServiceAccount (which is living in the customers namespace). Optimally this would be default as every K8s namespace is having this by default.

What keeps you from generating a VaultAuth for each namespace? Don't you already have a template for common namespace resources where this would fit neatly e.g. for RoleBindings etc.?

That is the alternative solution, yes. Until today we do not provision stuff in customers namespaces. We provide a Kustomize base and a Helm chart for Vault sync.

We used to include this so a platform customer could simply tick "Vault" when ordering a namespace and everything came already generated for them (both on k8s and vault side).

Sounds good, but most infra tools are operated in a different team inside our org:

• a team for Identity stuff
• a team for Security tooling (vuln scanning and HC Vault)
• a team for CI/CD (CI runners, GitOps tools)
• ... (you name it)