failed to create kubernetes rest client

[flashpixx]

Hello,

I try to deploy the ingress file https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml with the provider in this structure (this script below is a minimal example which can be executed)

terraform {
  required_version = ">= 0.13"

  required_providers {
    kind = {
      source  = "tehcyx/kind"
      version = ">= 0.0.12"
    }  
  
    http = {
      source  = "hashicorp/http"
      version = ">= 2.2.0"
    }

    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.14.0"
    }
  }
}

provider http {}

provider "kind" {}

provider "kubectl" {
  load_config_file       = false
  host                   = kind_cluster.k8s.endpoint
  client_certificate     = base64encode(kind_cluster.k8s.client_certificate)
  cluster_ca_certificate = base64encode(kind_cluster.k8s.cluster_ca_certificate)
}

resource "kind_cluster" "k8s" {
    name = "test-cluster"
}

data "http" "ingress" {
  url = "https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml"
}

data "kubectl_file_documents" "ingress" {
  content = data.http.ingress.response_body
}

resource "kubectl_manifest" "ingress" {
  for_each         = data.kubectl_file_documents.ingress.manifests
  yaml_body        = each.value
  wait_for_rollout = false
  force_new        = true
}

I get on a terraform apply the error log

kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-create"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-patch"]: Creating...
kubectl_manifest.ingress["/apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller"]: Creating...
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-create"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-patch"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-patch"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-create"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/networking.k8s.io/v1/ingressclasses/nginx"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-admission"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller"]: Creating...
kubectl_manifest.ingress["/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/ingress-nginx-admission"]: Creating...
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx"]: Creating...
kubectl_manifest.ingress["/apis/networking.k8s.io/v1/ingressclasses/nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/ingress-nginx-admission"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx"]: Still creating... [10s elapsed]
kubectl_manifest.ingress["/apis/networking.k8s.io/v1/ingressclasses/nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/ingress-nginx-admission"]: Still creating... [20s elapsed]
kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx"]: Still creating... [20s elapsed]
╷
│ Error: ingress-nginx/ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-admission-create failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-create"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/networking.k8s.io/v1/ingressclasses/nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/serviceaccounts/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-admission-patch failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/batch/v1/namespaces/ingress-nginx/jobs/ingress-nginx-admission-patch"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/roles/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-controller failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/apps/v1/namespaces/ingress-nginx/deployments/ingress-nginx-controller"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-controller-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/clusterroles/ingress-nginx"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-controller failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/services/ingress-nginx-controller"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-controller failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/api/v1/namespaces/ingress-nginx/configmaps/ingress-nginx-controller"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {
│ 
╵
╷
│ Error: ingress-nginx/ingress-nginx-admission failed to create kubernetes rest client for update of resource: an error on the server ("") has prevented the request from succeeding
│ 
│   with kubectl_manifest.ingress["/apis/rbac.authorization.k8s.io/v1/namespaces/ingress-nginx/rolebindings/ingress-nginx-admission"],
│   on cluster.tf line 53, in resource "kubectl_manifest" "ingress":
│   53: resource "kubectl_manifest" "ingress" {

How can I fix this? A normal command-line call with kubectl -f https://.. works fine and produces a valid output / configuration

[DaleyKD]

I have no idea if this is helpful or not, but this is our provider declaration:

provider "kubectl" {
  host                   = module.aks_cluster.kube_config.host
  client_certificate     = base64decode(module.aks_cluster.kube_config.client_certificate)
  client_key             = base64decode(module.aks_cluster.kube_config.client_key)
  cluster_ca_certificate = base64decode(module.aks_cluster.kube_config.cluster_ca_certificate)
  load_config_file       = false
}

Is client_key required?

[wscourge]

I'm getting the same error, with different reason:

Error: dev/hix-bg failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp 127.0.0.1:80: connect: connection refused

[DaleyKD]

I'm getting the same error, with different reason:

Error: dev/hix-bg failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp 127.0.0.1:80: connect: connection refused

Is your host supposed to be localhost? If not, is it set in the provider config? Finally, did you set load_config_file = false ?

[wscourge]

Yes, setting load_config_file = false solved this.

[maharshirt]

facing same issue and load_config_file = false this already in place but still getting error like below

Error: wasmplugins.extensions.istio.io failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused 2023-04-14T02:41:32.9556305Z 2023-04-14T02:41:32.9556673Z with module.central_region.module.istio_dsy.kubectl_manifest.istio_crds["/apis/apiextensions.k8s.io/v1/customresourcedefinitions/wasmplugins.extensions.istio.io"], 2023-04-14T02:41:32.9557292Z on .terraform/modules/central_region.istio_dsy/istio/base/istio.tf line 9, in resource "kubectl_manifest" "istio_crds": 2023-04-14T02:41:32.9557593Z 9: resource "kubectl_manifest" "istio_crds" { 2023-04-14T02:41:32.9557650Z 2023-04-14T02:41:32.9561037Z 2023-04-14T02:41:32.9562726Z Error: telemetries.telemetry.istio.io failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused 2023-04-14T02:41:32.9562882Z 2023-04-14T02:41:32.9563224Z with module.central_region.module.istio_dsy.kubectl_manifest.istio_crds["/apis/apiextensions.k8s.io/v1/customresourcedefinitions/telemetries.telemetry.istio.io"], 2023-04-14T02:41:32.9563455Z on .terraform/modules/central_region.istio_dsy/istio/base/istio.tf line 9, in resource "kubectl_manifest" "istio_crds": 2023-04-14T02:41:32.9563672Z 9: resource "kubectl_manifest" "istio_crds" { 2023-04-14T02:41:32.9563728Z 2023-04-14T02:41:32.9566926Z 2023-04-14T02:41:32.9567789Z Error: peerauthentications.security.istio.io failed to create kubernetes rest client for update of resource: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused 2023-04-14T02:41:32.9569034Z 2023-04-14T02:41:32.9569671Z with module.central_region.module.istio_dsy.kubectl_manifest.istio_crds["/apis/apiextensions.k8s.io/v1/customresourcedefinitions/peerauthentications.security.istio.io"], 2023-04-14T02:41:32.9570140Z on .terraform/modules/central_region.istio_dsy/istio/base/istio.tf line 9, in resource "kubectl_manifest" "istio_crds": 2023-04-14T02:41:32.9570496Z 9: resource "kubectl_manifest" "istio_crds" {

[valeriano-manassero]

I had same issue of @flashpixx and I found a solution, even if I'm not 100% sure on cause, I'll try to explain.

I had this config for kind provider:

resource "kind_cluster" "main_cluster" {
    name            = var.main_cluster_name
    kubeconfig_path = pathexpand(var.kubernetes_config_file)
}

and this one:

provider "kubectl" {
  load_config_file  = true
  config_path       = pathexpand(var.kubernetes_config_file)
}

After failing I applied again with the already existing kind cluster (and config file ofc), it worked.

So after some tests I changed config to this:

provider "kubectl" {
  load_config_file  = true
  config_path       = kind_cluster.main_cluster.kubeconfig_path
}

and it worked from scratch; for some reason, declaring the config file to use as a variable and not as output by kind provider was the problem.

Just for the records, these are my versions:

terraform {
  required_providers {
    kind = {
      source  = "tehcyx/kind"
      version = "~> 0.0.17"
    }
    http = {
      source  = "hashicorp/http"
      version = "~> 3.3.0"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "~> 1.14.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.9.0"
    }
  }
  required_version = "~> 1.4.0"
}

For more info, the repo I'm working on is: https://github.com/valeriano-manassero/kubernetes-gitops-dev