Waiting for a custom resource to complete and proceeding

[linuxbsdfreak]

Hi @gavinbunney ,

I am using the kubectl provider to install a K8s Custom Resource. The issue is that the provider does a kubectl apply and proceeds ahead. Is there a way to wait until the Custom Resource gets reconciled by polling a lastoperation field or any other field in the Custom Resource.

I am doing the following on the command line

k get shoots testcluster -o yaml

I have the following states that i could poll

 lastOperation:
    description: Waiting until shoot worker nodes have been reconciled
    lastUpdateTime: "2020-09-03T11:14:18Z"
    progress: 82
    state: Processing
    type: Reconcile

  lastOperation:
    description: Shoot cluster state has been successfully reconciled.
    lastUpdateTime: "2020-09-03T11:17:23Z"
    progress: 100
    state: Succeeded
    type: Reconcile

I would like to wait until the state is Succeeded and then go head for other terraform steps. Is that possible to have something like that the provider ?

I also saw https://github.com/gavinbunney/terraform-provider-kubectl/issues/31.

Something like a polling with kubectl and getting lastoperation until it succeeded would be helpful.

Kevin

[vrabbi]

This would be great. A similar idea exists in the kubernetes alpha provider which allows the selection of what condition to wait for. Having that here would be a huge improvement

[linuxbsdfreak]

This would be great. A similar idea exists in the kubernetes alpha provider which allows the selection of what condition to wait for. Having that here would be a huge improvement

Thanks @vrabbi 👍 . I have created a small work around with executing a golang binary to parse the status filed with the null_resource in TF. It is a little hackish. However native integration similar to https://github.com/hashicorp/terraform-provider-kubernetes-alpha#using-wait_for-to-block-create-and-update-calls would help.

[a0s]

@linuxbsdfreak Some time ago i am tried to resolve the same issue (waiting for CRD). I wrote https://github.com/a0s/terraform-provider-kubernetes-awaiter which accepts CRD resource as URI and waiting until it appear.

[linuxbsdfreak]

@a0s Thanks for the info. How would one wait for the Status in the CR field. ? This is what the official TF K8s provider has

  wait_for = {
    fields = {
      # Check the phase of the CR
      "status.lastOperation.state" = "Succeeded"
    }
  }

[linuxbsdfreak]

@gavinbunney Do you think we can query the CR for providing the status ? I am missing this feature for doing the automation with Terraform

[jimmy-scott]

Hello @gavinbunney , I have code in progress for both:

wait_for_attrs = {
  "status.phase" = "Succeeded"
}

and:

wait_for_conditions = {
  "Ready" = "True"
}

Which waits for all conditions to be met (one at a time).

Currently wait_for_attrs works for string values only, but I can look into improving that. Also waiting time still needs to be recalculated between checking waiting conditions, but I can add that as well.

Let me know if you would be interested in merging these features or how you would like to see them. I can send it as a patch or setup a fork and create a pull request .. let me know how you like to proceed.

[linuxbsdfreak]

Hi @jimmy-scott

That’s nice. Once merged in the provider. Can I do something like this

wait_for_attrs= { # Check the phase of the CR "status.lastOperation.state" = "Succeeded" }

[jimmy-scott]

That’s nice. Once merged in the provider. Can I do something like this

wait_for_attrs= { # Check the phase of the CR "status.lastOperation.state" = "Succeeded" }

Correct! It works at any depth.

[linuxbsdfreak]

Thx @jimmy-scott . Can we get it merged for testing?

[linuxbsdfreak]

@jimmy-scott @gavinbunney Could we please have a new provider with this fix ?

[linuxbsdfreak]

@jimmy-scott Do you have a fork of the provider at your end that i can use for testing ?

[jimmy-scott]

Hi @linuxbsdfreak , I was hoping @gavinbunney could give me some advice on how to proceed… ideally I just wanted to send the patches by mail. But since I didn't got any response and already working on something else now, I can't spend too much time on this anymore. I think I will be putting the forked code online on a separate github account and inform you once available. This can take some time though.

[linuxbsdfreak]

Hi @gavinbunney . Would it be possible to merge the changes from @jimmy-scott ?

[linuxbsdfreak]

@jimmy-scott Could you please share the changes and also the process for creating a new provider with the changes ? Would have loved to have it merged in the kubectl provider. I assume @gavinbunney is not reachable.

[linuxbsdfreak]

@jimmy-scott Could you provide me the changes and the process to create the provider?

[jimmy-scott]

@jimmy-scott Could you provide me the changes and the process to create the provider?

If you can send me an email, I can already send you a patch with ALL the changes I’ve made. Would that help?

My email address can be found in source files from my repositories in the copyright line of the license.

[linuxbsdfreak]

@jimmy-scott . You can send me an email to [email protected]. Could you also let me know the process to compile the provider and testing it locally. Thx

[linuxbsdfreak]

@gavinbunney Would it be possible to merge the changes from @jimmy-scott ? It would save a lot of work arounds with running kubectl .

I am doing the following

resource "null_resource" "check_shoot_status" {
  triggers = {
    always_run = "${timestamp()}"
  }

  provisioner "local-exec" {
    command = "while [ true ]; do STATUS=`kubectl --kubeconfig ${var.robot_local_path} get Shoot ${var.cluster_name} -ojsonpath='{.status.lastOperation.state}'`; if [ \"$STATUS\" = \"Succeeded\" ]; then echo \"SUCCEEDED\" ; break ; else echo \"INPROGRESS\"; fi ; done"
    interpreter = ["/bin/bash", "-c"]
  }

  depends_on = [kubectl_manifest.gardener_shoot]
}

[alekc]

Just fiy, this is merged and deployed on my fork, https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/kubectl_manifest#with-explicit-wait_for I.e.

resource "kubectl_manifest" "test" {
  wait_for {
    field {
      key = "status.containerStatuses.[0].ready"
      value = "true"
    }
    field {
      key = "status.phase"
      value = "Running"
    }
    field {
      key = "status.podIP"
      value = "^(\\d+(\\.|$)){4}"
      value_type = "regex"
    }
  }
  yaml_body = <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    readinessProbe:
      httpGet:
        path: "/"
        port: 80
      initialDelaySeconds: 10
YAML
}

terraform {
  required_providers {
    kubectl = {
      source = "alekc/kubectl"
      version = "2.0.2"
    }
  }
}

[joaocc]

Thanks. Do you know if we can expect status.conditions[] to always have the same types in each positions (for a specific manifest type)? I'm asking as I was trying to apply wait_for to the example below, in order to detect when conditions "Ready" and "Released" were set to "True".

status:
  conditions:
    - lastTransitionTime: '2023-10-31T13:25:06Z'
      message: Release reconciliation succeeded
      reason: ReconciliationSucceeded
      status: 'True'
      type: Ready
    - lastTransitionTime: '2023-10-31T13:25:06Z'
      message: Helm install succeeded
      reason: InstallSucceeded
      status: 'True'
      type: Released
  helmChart: flux-system/some-helm-release
  lastAppliedRevision: 1.1.1
  lastAttemptedRevision: 1.1.1
  lastReleaseRevision: 1
  observedGeneration: 1

Since wait_for only seems to provide the ability to access status.conditions via numeric indexing (as array), I was wondering if anything in kubernetes spec for conditions can provide the answer for that (https://maelvls.dev/kubernetes-conditions/ is quite complete description, but I could not find the answer there) Thanks