terraform-google-kubernetes-engine
terraform-google-kubernetes-engine copied to clipboard
terraform-google-modules
Using exisiting gcp sa and k8s sa the null object command to anontate fails.
TL;DR
When attempting to use this module with and existing SA's the module's null object command for kubectl anontate fails the command succeeds on the command line. Claim annotate command not found.
Expected behavior
k8s SA is annotated with gcp SA.
Observed behavior
[email protected]/roles/iam.workloadIdentityUser/serviceAccount:PROJECT.svc.id.goog[iac-cicd/iac-cicd-gitlab-runner]] ╷ │ Error: local-exec provisioner error │ │ with module.cicd_bootstrap.module.my-app-workload-identity.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0], │ on .terraform/modules/cicd_bootstrap.my-app-workload-identity.annotate-sa/main.tf line 231, in resource "null_resource" "run_command": │ 231: provisioner "local-exec" { │ │ Error running command 'PATH=/google-cloud-sdk/bin:$PATH │ .terraform/modules/cicd_bootstrap.my-app-workload-identity.annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh PROJECT false false kubectl annotate │ --overwrite sa -n iac-cicd iac-cicd-gitlab-runner iam.gke.io/gcp-service-account=org-cicd-account@PROJECT.iam.gserviceaccount.com │ ': exit status 1. Output: + '[' 11 -lt 5 ']' │ + CLUSTER_NAME=CLUSTERNAME │ + LOCATION=false │ + PROJECT_ID=false │ + INTERNAL=kubectl │ + USE_EXISTING_CONTEXT=annotate │ + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=--overwrite │ + IMPERSONATE_SERVICE_ACCOUNT=sa │ + shift 5 │ + annotate │ .terraform/modules/cicd_bootstrap.my-app-workload-identity.annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh: line 34: annotate: command not found │ + RANDOM_ID=24686_25245 │ + export TMPDIR=/tmp/kubectl_wrapper_24686_25245 │ + TMPDIR=/tmp/kubectl_wrapper_24686_25245 │ + trap cleanup EXIT │ + mkdir /tmp/kubectl_wrapper_24686_25245 │ + export KUBECONFIG=/tmp/kubectl_wrapper_24686_25245/config │ + KUBECONFIG=/tmp/kubectl_wrapper_24686_25245/config │ ++ grep -o - │ ++ wc -l │ + LOCATION_TYPE=' 0' │ + cleanup │ + rm -rf /tmp/kubectl_wrapper_24686_25245
Terraform Configuration
module "my-app-workload-identity" {
source = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
version = "17.1.0"
use_existing_k8s_sa = true
use_existing_gcp_sa = true
name = "[email protected]"
gcp_sa_name = "[email protected]"
k8s_sa_name = "iac-cicd-gitlab-runner"
namespace = "iac-cicd"
project_id = module.cicd_project.project_id
annotate_k8s_sa = true
automount_service_account_token = true
}
Terraform Version
iac-cicd % terraform --version
Terraform v1.0.7
on darwin_amd64
+ provider registry.terraform.io/gitlabhq/gitlab v3.7.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/google v3.85.0
+ provider registry.terraform.io/hashicorp/google-beta v3.85.0
+ provider registry.terraform.io/hashicorp/helm v2.3.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.5.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.1
+ provider registry.terraform.io/hashicorp/template v2.2.0
Your version of Terraform is out of date! The latest version
is 1.0.11. You can update by downloading from https://www.terraform.io/downloads.html
Additional information
No response
I can reproduce this bug even if use_existing_gcp_sa is false.
The output command works :
kubectl annotate --overwrite sa -n iac-cicd iac-cicd-gitlab-runner iam.gke.io/gcp-service-account=org-cicd-account@PROJECT.iam.gserviceaccount.com
The interesting part from the log: annotate: command not found. I would bet on the shift 5 removing the kubectl string from the exec command string
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0]: Creating...
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0]: Provisioning with 'local-exec'...
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n.terraform/modules/workload-identity-cert-manager.annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh devops myproject false false kubectl annotate --overwrite sa -n cert-manager cert-manager iam.gke.io/gcp-service-account=dns01-solver@myproject.iam.gserviceaccount.com\n"]
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + '[' 12 -lt 5 ']'
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + CLUSTER_NAME=devops
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + LOCATION=myproject
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + PROJECT_ID=false
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + INTERNAL=false
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + USE_EXISTING_CONTEXT=kubectl
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=annotate
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + IMPERSONATE_SERVICE_ACCOUNT=--overwrite
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + shift 5
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + kubectl
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kubectl controls the Kubernetes cluster manager.
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Basic Commands (Beginner):
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): create Create a resource from a file or from stdin
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes service
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): run Run a particular image on the cluster
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): set Set specific features on objects
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Basic Commands (Intermediate):
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): explain Get documentation for a resource
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): get Display one or many resources
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): edit Edit a resource on the server
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): delete Delete resources by file names, stdin, resources and names, or by resources and label selector
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Deploy Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): rollout Manage the rollout of a resource
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): scale Set a new size for a deployment, replica set, or replication controller
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): autoscale Auto-scale a deployment, replica set, stateful set, or replication controller
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Cluster Management Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): certificate Modify certificate resources.
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cluster-info Display cluster information
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): top Display resource (CPU/memory) usage
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cordon Mark node as unschedulable
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): uncordon Mark node as schedulable
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): drain Drain node in preparation for maintenance
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): taint Update the taints on one or more nodes
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Troubleshooting and Debugging Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): describe Show details of a specific resource or group of resources
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): logs Print the logs for a container in a pod
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): attach Attach to a running container
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): exec Execute a command in a container
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): port-forward Forward one or more local ports to a pod
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): proxy Run a proxy to the Kubernetes API server
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cp Copy files and directories to and from containers
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): auth Inspect authorization
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): debug Create debugging sessions for troubleshooting workloads and nodes
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Advanced Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): diff Diff the live version against a would-be applied version
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): apply Apply a configuration to a resource by file name or stdin
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): patch Update fields of a resource
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): replace Replace a resource by file name or stdin
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): wait Experimental: Wait for a specific condition on one or many resources
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kustomize Build a kustomization target from a directory or URL.
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Settings Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): label Update the labels on a resource
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): annotate Update the annotations on a resource
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): completion Output shell completion code for the specified shell (bash or zsh)
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Other Commands:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): api-resources Print the supported API resources on the server
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): api-versions Print the supported API versions on the server, in the form of "group/version"
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): config Modify kubeconfig files
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): plugin Provides utilities for interacting with plugins
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): version Print the client and server version information
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Usage:
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kubectl [flags] [options]
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Use "kubectl <command> --help" for more information about a given command.
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Use "kubectl options" for a list of global command-line options (applies to all commands).
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + annotate --overwrite sa -n cert-manager cert-manager iam.gke.io/gcp-service-account=dns01-solver@myproject.iam.gserviceaccount.com
module.workload-identity-cert-manager.module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): .terraform/modules/workload-identity-cert-manager.annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh: line 36: annotate: command not found
I was getting this exact same issue, and it was because there are a few fields marked as
Required if using existing KSA.
For me I had to set cluster_name, and location to get this working.
I also ended up needing to manually remove some of these that were partially applied from the state
terraform state rm blah.module.annotate-sa.module.gcloud_kubectl.null_resource.run_destroy_command[0] since the destroy command was cached in the state so even after I set cluster_name and location, it was running the cached command that was failing.
Hope this helps.
Came across this exact error, and those three steps (cluster_name, location, state rm) worked for me!
It would be good to add some validation rules to these two vars to force the requirement instead of failing pathologically.
It would be good to add some validation rules to these two vars to force the requirement instead of failing pathologically.
Great idea, would be happy to review a PR.
Or at least call it out in the docs clearly.
On Thu, Jan 20, 2022 at 12:32 PM lincoln-replit @.***> wrote:
Came across this exact error, and those three steps (cluster_name, location, state rm) worked for me!
It would be good to add some validation rules https://www.terraform.io/language/values/variables#custom-validation-rules to these two vars to force the requirement instead of failing pathologically.
— Reply to this email directly, view it on GitHub https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/issues/1065#issuecomment-1017904150, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYOOKYKTL4UWJDPSWSHHLDUXBWOJANCNFSM5IFRH5FA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: <terraform-google-modules/terraform-google-kubernetes-engine/issues/1065/1017904150 @github.com>