ASM 1.10 script is failing to validate when istio-system namespace doesn't exist.

[giulianobr]

Hi there, it seems the #638 is back in the 1.10 script installation. I'm doing a new setup with ASM 1.10 + terraform and failing because of the same describe in the #638 Would be nice to have the fix merged in the 1.10 script.

Thanks,

[zerobfd]

Hello! Thanks for reporting.

We took a preliminary look at this yesterday and the commit that fixed it in previous versions is also in 1.10, so we're still looking to see what could be causing it. If you have any output from the TF run it would be useful to help debug.

[zerobfd]

Oh, does this mean that using --only-validate fails?

[giulianobr]

Hi there, sorry my delay. Tomorrow I'll do a new installation, so I can provide more details. Thanks!

[giulianobr]

Hi all, here is the error log:

module.asm-gke.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + rm -rf /tmp/kubectl_wrapper_27061_5623

Error: local-exec provisioner error

  with module.asm-gke.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0],
  on .terraform/modules/asm-gke.asm_install/main.tf line 231, in resource "null_resource" "run_command":
 231:   provisioner "local-exec" {

Error running command 'PATH=/google-cloud-sdk/bin:$PATH
.terraform/modules/asm-gke.asm_install/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh MY_GKE_CLUSTER_ID europe-west1 MY_PROJECT_ID false false
.terraform/modules/asm-gke/modules/asm/scripts/install_asm.sh MY_PROJECT_ID MY_GKE_CLUSTER_ID europe-west1 1.10 install false false cloud-tracing
./ingress-backendconfig-operator.yaml false true true true true none meshca none none none none tf-local@MY_PROJECT_ID.iam.gserviceaccount.com sa.json none
': exit status 2. Output: ARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1
install_asm_1.10: Checking Istio installations...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -A --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -n istio-system --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -n istio-system --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud services list --enabled --format=get(config.name) --project=MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Checking required APIs...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig api-resources --api-group=hub.gke.io'
install_asm_1.10: -------------
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe --project=MY_PROJECT_ID --region europe-west1
MY_GKE_CLUSTER_ID --format=json'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe MY_GKE_CLUSTER_ID --zone=europe-west1
--project=MY_PROJECT_ID --format=value(selfLink, network)'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container hub memberships list --format=value(name) --project MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container hub memberships list --format=value(name) --project MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Registering the cluster as MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud beta container hub memberships register MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--gke-uri=https://container.googleapis.com/v1/projects/MY_PROJECT_ID/locations/europe-west1/clusters/MY_GKE_CLUSTER_ID --enable-workload-identity'
install_asm_1.10: -------------
kubeconfig entry generated for MY_GKE_CLUSTER_ID.
Waiting for membership to be created...
.......................done.
Created a new membership [projects/MY_PROJECT_ID/locations/global/memberships/MY_GKE_CLUSTER_ID] for the cluster [MY_GKE_CLUSTER_ID]
Generating the Connect Agent manifest...
Deploying the Connect Agent on cluster [MY_GKE_CLUSTER_ID] in namespace [gke-connect]...
Deployed the Connect Agent on cluster [MY_GKE_CLUSTER_ID] in namespace [gke-connect].
Finished registering the cluster [MY_GKE_CLUSTER_ID] with the Hub.
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud projects get-iam-policy MY_PROJECT_ID --flatten=bindings[].members
--filter=bindings.members:serviceAccount:tf-local@MY_PROJECT_ID.iam.gserviceaccount.com --format=value(bindings.role)'
install_asm_1.10: -------------
install_asm_1.10: Checking for project MY_PROJECT_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud projects describe MY_PROJECT_ID --format=value(projectNumber)'
install_asm_1.10: -------------
install_asm_1.10: Reading labels for europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe MY_GKE_CLUSTER_ID --zone=europe-west1
--project=MY_PROJECT_ID --format=value(resourceLabels)[delimiter=","]'
install_asm_1.10: -------------
install_asm_1.10: Adding labels to europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters update MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--zone=europe-west1 --update-labels=asmv=1-10-2-asm-3,mesh_id=proj-978270309481'
install_asm_1.10: -------------
Updating MY_GKE_CLUSTER_ID...
.........................done.
Updated [https://container.googleapis.com/v1/projects/MY_PROJECT_ID/zones/europe-west1/clusters/MY_GKE_CLUSTER_ID].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1/MY_GKE_CLUSTER_ID?project=MY_PROJECT_ID
install_asm_1.10: Initializing meshconfig API...
install_asm_1.10: Running: 'curl --request POST --fail --data  -o /dev/null https://meshconfig.googleapis.com/v1alpha1/projects/MY_PROJECT_ID:initialize -K /dev/fd/63'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud --project=MY_PROJECT_ID auth print-access-token'
install_asm_1.10: -------------
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     3    0     3    0     0      4      0 --:--:-- --:--:-- --:--:--     4
install_asm_1.10: Enabling Stackdriver on europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters update MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--zone=europe-west1 --enable-stackdriver-kubernetes'
install_asm_1.10: -------------
Updating MY_GKE_CLUSTER_ID...
........................done.
Updated [https://container.googleapis.com/v1/projects/MY_PROJECT_ID/zones/europe-west1/clusters/MY_GKE_CLUSTER_ID].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1/MY_GKE_CLUSTER_ID?project=MY_PROJECT_ID
install_asm_1.10: Querying for core/account...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud config get-value core/account'
install_asm_1.10: -------------
Your active configuration is: [devoteam]
install_asm_1.10: Binding tf-local@MY_PROJECT_ID.iam.gserviceaccount.com to cluster admin role...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig create clusterrolebinding tf-local-cluster-admin-binding --clusterrole=cluster-admin
--user=tf-local@MY_PROJECT_ID.iam.gserviceaccount.com --dry-run -o yaml'
install_asm_1.10: -------------
W0728 09:55:27.024740   10009 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig apply -f -'
install_asm_1.10: -------------
clusterrolebinding.rbac.authorization.k8s.io/tf-local-cluster-admin-binding created
install_asm_1.10: Checking for istio-system namespace...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get ns'
install_asm_1.10: -------------
install_asm_1.10: [ERROR]: The istio-system namespace doesn't exist.
Please create the "istio-namespace" and retry, or run the script with the
'--enable_namespace_creation' flag to allow the script to enable it on your behalf.
Alternatively, use --enable_all|-e to allow this tool to handle all dependencies.
+ cleanup
+ rm -rf /tmp/kubectl_wrapper_27061_5623

[giulianobr]

That service account has these roles:

ROLE roles/container.admin roles/editor roles/gkehub.admin roles/gkehub.gatewayAdmin roles/gkehub.viewer roles/iam.serviceAccountAdmin roles/logging.logWriter roles/meshconfig.admin roles/monitoring.metricWriter roles/serviceusage.serviceUsageConsumer

[giulianobr]

The ASM module:

module "asm-gke" {
  source                = "terraform-google-modules/kubernetes-engine/google//modules/asm"
  version               = "15.0.1"
  asm_version           = var.asm_version
  project_id            = var.project_id
  cluster_name          = module.gke.name
  location              = module.gke.location
  cluster_endpoint      = module.gke.endpoint
  enable_all            = false
  enable_gcp_apis       = true
  enable_gcp_components = true
  enable_cluster_labels = true
  enable_cluster_roles  = true
  enable_registration   = true
  enable_gcp_iam_roles  = false
  options               = ["cloud-tracing"]
  custom_overlays       = ["./ingress-backendconfig-operator.yaml"]
}

[ZhengzheYang]

add enable_namespace_creation flag and start testing with 1.10 #968 is merged. Please try with an additional flag enable_namespace_creation = true in the ASM module. Let me know if it does not work. Thanks!