"secrets <...> not found" when creating service account token

[jamshaidishaque]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: Terraform v0.14.7
Kubernetes provider version: 2.18.0
Kubernetes version: 1.24.9

Terraform Configuration Files

  metadata {
    name      = "${var.service_account_name}-token-secret"
    namespace = var.service_account_namespace
    annotations = {
      "kubernetes.io/service-account.name"      = var.service_account_name
      "kubernetes.io/service-account.namespace" = var.service_account_namespace
    }
  }
  data = {
    token = var.crm_service_account_token
  }
  type = "kubernetes.io/service-account-token"
}

resource "kubernetes_service_account" "service_account" {
  automount_service_account_token = false

  metadata {
    name      = var.service_account_name
    namespace = var.service_account_namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${aws_iam_role.external-secrets.name}"
    }
  }
  secret {
    name = "${kubernetes_secret.service_account_secret.metadata.0.name}"
  }
}

Steps to Reproduce

1. terraform apply -->

Expected Behavior

Service account token should be generated

Actual Behavior

secrets "" not found

References

https://github.com/hashicorp/terraform-provider-kubernetes/issues/1879 https://github.com/hashicorp/terraform-provider-kubernetes/issues/254

[anden-dev]

The documented example has this problem as well:

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret_v1#example-usage-service-account-token

resource "kubernetes_secret_v1" "example" {
  metadata {
    annotations = {
      "kubernetes.io/service-account.name" = "my-service-account"
    }
  }

  type = "kubernetes.io/service-account-token"
}

Terraform version: Terraform  1.3.8
Kubernetes provider version: 2.18.0
Kubernetes version: v1.24.8 (eks)

To replicate

resource "kubernetes_service_account" "eks_admin" {
  metadata {
    name      = "eks-admin"
    namespace = "kube-system"

  }
  automount_service_account_token = true

  lifecycle {
    ignore_changes = [
      secret
    ]
  }
}

resource "kubernetes_secret_v1" "eks_admin" {
  metadata {
    annotations = {
      "kubernetes.io/service-account.name" = kubernetes_service_account.eks_admin.metadata.0.name
    }
  }

  type = "kubernetes.io/service-account-token"
}

[alexsomesan]

There is some serious strangeness around the current API behaviour when it comes to service account token secrets. It seems that the API requires the SA to already be present when attempting to create a token secret that references it. However, it does not seem to check for it during the processing of the POST call that creates the secret. Instead, the API accepts the POST call and confirms the creation of the secret with a 201 response, but then silently drops it and subsequent GET calls predictably fail with "404 not found".

In light of this, it seems that the only course of action is to make sure that the SA gets created first and then the secret. To make sure Terraform performs its actions in this order, you have to define a dependency between the SA and the secret, by referencing an attribute of the SA resource in the secret resource. In the example above, this produces a cyclic dependency due to the name attribute of the secret being referenced in the "secret" attribute of the SA. So to break the cycle, you can use a local variable to compose the name of the secret and then pass that into both the secret and the SA resource.

Here's how I refactored the example above to get it to work:

locals {
  secret_name = "${var.service_account_name}-token-secret"
}

resource "kubernetes_service_account" "service_account" {
  automount_service_account_token = false

  metadata {
    name      = var.service_account_name
    namespace = var.service_account_namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = "arn:aws:iam::debug"
    }
  }
  secret {
    name = local.secret_name
  }
}

resource "kubernetes_secret" "service_account_secret" {
  metadata {
    name      = local.secret_name
    namespace = var.service_account_namespace
    annotations = {
      "kubernetes.io/service-account.name"      = kubernetes_service_account.service_account.metadata.0.name
      "kubernetes.io/service-account.namespace" = var.service_account_namespace
    }
  } 
  type = "kubernetes.io/service-account-token"
  data = {
    token = var.crm_service_account_token
  }
}

I would also encourage everyone to use the *_v1 variant of these resources (kubernetes_service_account_v1 and kubernetes_secret_v1), as these are guaranteed to use the stable API version as opposed to the beta one.

Let me know if this resolves your issues.

[varunthakur2480]

I have got a similar issue, have tried everything from adding dependency to using v1 version for creation of SA and the secret nothing works and it always fails at same point while trying to create the secret Error: secrets XXX" not found

[varunthakur2480]

I have manage to resolve it now . It turned out the api call for service account was creating the secret in default namespace as I had forgotten to specify it explicitly. However after creating the secret it was silently failing to attach it the service account and eventually deleting it without throwing any error. Later on it tried running another get for secret and failed with secret not found error Certainly needs better error handling

[cpanixperi]

I am getting this Error: secrets not foundon terraform apply . It works on the second terraform apply. Not sure what is happening here.

[stigok]

Helpful related comment from @sheneska in issue https://github.com/hashicorp/terraform-provider-kubernetes/issues/2029:

Hi @varunvijayagopal, When it comes to service account token secrets, there are some discrepancies with the current API behavior. It appears that in order to attempt to build a token secret that references the SA, the API needs to be active. Yet, it doesn't appear to perform a check for it when handling the POST request that generates the secret. Instead, the API accepts the POST request and replies with a 201 code to confirm that the secret was created. However, after that, it drops the request silently, and any subsequent GET requests will inevitably fail with a 404 error.

Given this, it would appear that the only viable option is to ensure that the SA is formed first, followed by the secret. You must provide a dependency between the SA and the secret resource by referencing an attribute of the SA resource in the secret resource to ensure that Terraform executes its operations in this sequence. Due to the name attribute of the secret being referenced in the "secret" element of the SA in the aforementioned example, this results in a cyclic dependency. In order to break the cycle, you can create the secret's name using a local variable and then send it to both the secret and the SA resource.

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!