can obtain external load balancer IP with provider "kubernetes" and resource "kubernetes_ingress" but not with provider "kubernetes-alpha" and resource "kubernetes_manifest"

[mattduguid]

Versions

aks kubernetes v1.19.11 (azure kubernetes service)
terraform v0.15.1
kubernetes provider v2.3.2
kubernetes-alpha provider v0.5.0

Question

can obtain external load balancer IP with provider "kubernetes" and resource "kubernetes_ingress" but not with provider "kubernetes-alpha" and resource "kubernetes_manifest", is this expected at this stage in the alpha provider?

Working using,

provider = kubernetes
resource = kubernetes_ingress

we use "wait_for_load_balancer = true" in the resource and in the debug logs we see the following repeat, "Load Balancer not ready yet" and "Waiting 500ms before next try",
...
2021-06-22T20:07:37.8721602Z  "status": {
2021-06-22T20:07:37.8721887Z   "loadBalancer": {}
2021-06-22T20:07:37.8722189Z  }
2021-06-22T20:07:37.8722404Z }
2021-06-22T20:07:37.8722517Z 
2021-06-22T20:07:37.8723114Z -----------------------------------------------------: timestamp=2021-06-22T20:07:37.868Z
2021-06-22T20:07:37.8724123Z 2021-06-22T20:07:37.868Z [INFO]  provider.terraform-provider-kubernetes_v2.3.2_x5: 2021/06/22 20:07:37 [INFO] Load Balancer not ready yet...: timestamp=2021-06-22T20:07:37.868Z
2021-06-22T20:07:37.8725307Z 2021-06-22T20:07:37.868Z [INFO]  provider.terraform-provider-kubernetes_v2.3.2_x5: 2021/06/22 20:07:37 [TRACE] Waiting 500ms before next try: timestamp=2021-06-22T20:07:37.868Z
2021-06-22T20:07:38.3704526Z 2021-06-22T20:07:38.369Z [INFO]  provider.terraform-provider-kubernetes_v2.3.2_x5: 2021/06/22 20:07:38 [DEBUG] Kubernetes API Request Details:
...

and then finally this where we can obtain the external IP and write to DNS,
...
2021-06-22T20:08:23.5340042Z  "status": {
2021-06-22T20:08:23.5340320Z   "loadBalancer": {
2021-06-22T20:08:23.5340601Z    "ingress": [
2021-06-22T20:08:23.5340864Z     {
2021-06-22T20:08:23.5341131Z      "ip": "20.53.xxx.xxx"
2021-06-22T20:08:23.5341398Z     }
2021-06-22T20:08:23.5341613Z    ]
2021-06-22T20:08:23.5341838Z   }
2021-06-22T20:08:23.5342063Z  }
2021-06-22T20:08:23.5342284Z }
...

Not working using

provider = kubernetes-alpha
resource = kubernetes_manifest 

and reading this https://github.com/hashicorp/terraform-provider-kubernetes-alpha and extract "wait_for currently supports a fields attribute which allows you specify a map of fields paths to regular expressions. You can also specify * if you just want to wait for a field to have any value." then using block,

  wait_for = {
    fields = {
      "status.loadBalancer.ingress[0].ip" = "*" # wait for external IP
    }
  }

using kubectl query can see that the ingress and external IP have created pretty quickly,
...
kubectl -n helloworld get services
NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
helloworld                                      ClusterIP      10.0.74.0      <none>         80/TCP                       95s
ingress-nginx-helloworld-controller             LoadBalancer   10.0.238.146   20.53.xxx.xxx  80:32369/TCP,443:31604/TCP   119s
ingress-nginx-helloworld-controller-admission   ClusterIP      10.0.171.162   <none>         443/TCP                      119s
ingress-nginx-helloworld-controller-metrics     ClusterIP      10.0.35.180    <none>         10254/TCP                    119s
...

but the wait_for condition never meets as its looks like there is no status or external IP attribute and then we get stuck in this loop,
...
module.k8s-deployment-helloworld.kubernetes_manifest.ingress: Still creating... [5m10s elapsed]
module.k8s-deployment-helloworld.kubernetes_manifest.ingress: Still creating... [5m20s elapsed]
...

if we take the wait_for condition out it the ingress resource complete but we still dont see any status or external IP either in the debug output or terraform state that can be used for the condition? is this expected at this stage in the alpha provider?
...
2021-06-22T21:03:08.6568322Z [0m[1mmodule.k8s-deployment-helloworld.kubernetes_manifest.ingress: Creating...[0m[0m
2021-06-22T21:03:08.6569165Z 2021-06-22T21:03:08.640Z [INFO]  Starting apply for module.k8s-deployment-helloworld.kubernetes_manifest.ingress
2021-06-22T21:03:08.6570089Z 2021-06-22T21:03:08.641Z [DEBUG] module.k8s-deployment-helloworld.kubernetes_manifest.ingress: applying the planned Create change
2021-06-22T21:03:09.3612544Z [0m[1mmodule.k8s-deployment-helloworld.kubernetes_manifest.ingress: Creation complete after 0s[0m
2021-06-22T21:03:09.3650926Z 2021-06-22T21:03:09.361Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-06-22T21:03:09.3703951Z 2021-06-22T21:03:09.364Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes-alpha/0.5.0/linux_amd64/terraform-provider-kubernetes-alpha_v0.5.0_x5 pid=3434
...

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[mattduguid]

similar result with tests to obtain external IP for a "service" as we had with "ingress", the data is definitely query-able with kubectl just not returned via terraform, can extract it with null_resource as a workaround for testing

kubectl -n <NAMESPACE> get service <SERVICENAME> --output jsonpath='{.status.loadBalancer.ingress[0].ip}' 20.53.xxx.xxx

kubectl -n <NAMESPACE> get ingress <SERVICENAME> --output jsonpath='{.status.loadBalancer.ingress[0].ip}' 20.53.xxx.xxx