`terraform destroy` command on a Kubernetes cluster doesn't remove Droplets in its Nodepool

[uniglot]

Bug Report

---

Describe the bug

Issuing a terraform destroy command on a "digitalocean_kubernetes_cluster" resource which doesn't have additional Nodepools removes the cluster, but the Droplets in the default Nodepool were not removed.

I found this behavior from the fact that creating the cluster took 5m30s but for destroying it 1s was enough.

After checking out the DO console, I found out that the Droplets were not removed and still running.

Affected Resource(s)

• digitalocean_kubernetes_cluster

Expected Behavior

Upon issuing a terraform destory command on the cluster's resource, I expected the Droplets which were in the default Nodepool should be removed along with the cluster.

Actual Behavior

The Droplets in the default Nodepool were not removed and still running.

Steps to Reproduce

1. terraform apply including the resource declaration below:

resource "digitalocean_kubernetes_cluster" "cluster" {
  name    = "uniglot-cluster-abc"
  region  = "sgp1"
  version = "1.26.3-do.0"

  node_pool {
    name       = "worker-pool"
    size       = "s-1vcpu-2gb"
    auto_scale = true
    min_nodes  = 3
    max_nodes  = 5
  }
}

which should work well.

2. terraform destroy and the generated plan will be similar to:

digitalocean_kubernetes_cluster.cluster: Refreshing state... [id=c8beca30-502d-4d4d-807c-256bedff2e91]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # digitalocean_kubernetes_cluster.cluster will be destroyed
  - resource "digitalocean_kubernetes_cluster" "cluster" {
      - auto_upgrade         = false -> null
      - cluster_subnet       = "10.244.0.0/16" -> null
      - created_at           = "2023-05-19 06:38:28 +0000 UTC" -> null
      - endpoint             = "https://c8beca30-502d-4d4d-807c-256bedff2e91.k8s.ondigitalocean.com" -> null
      - ha                   = false -> null
      - id                   = "c8beca30-502d-4d4d-807c-256bedff2e91" -> null
      - kube_config          = (sensitive value) -> null
      - name                 = "uniglot-cluster-abc" -> null
      - region               = "sgp1" -> null
      - registry_integration = false -> null
      - service_subnet       = "10.245.0.0/16" -> null
      - status               = "running" -> null
      - surge_upgrade        = true -> null
      - tags                 = [] -> null
      - updated_at           = "2023-05-19 06:43:57 +0000 UTC" -> null
      - urn                  = "do:kubernetes:c8beca30-502d-4d4d-807c-256bedff2e91" -> null
      - version              = "1.26.3-do.0" -> null
      - vpc_uuid             = "f0a6ba4a-011a-4fa7-ac8e-57add95e8eca" -> null

      - maintenance_policy {
          - day        = "any" -> null
          - duration   = "4h0m0s" -> null
          - start_time = "0:00" -> null
        }

      - node_pool {
          - actual_node_count = 3 -> null
          - auto_scale        = true -> null
          - id                = "94673e6d-066f-4626-bdde-6a546fe41f93" -> null
          - labels            = {} -> null
          - max_nodes         = 5 -> null
          - min_nodes         = 3 -> null
          - name              = "worker-pool" -> null
          - node_count        = 0 -> null
          - nodes             = [
              - {
                  - created_at = "2023-05-19 06:38:28 +0000 UTC"
                  - droplet_id = "356101631"
                  - id         = "6003a756-baa9-4b25-a962-8027fd529f2c"
                  - name       = "worker-pool-frffl"
                  - status     = "running"
                  - updated_at = "2023-05-19 06:40:39 +0000 UTC"
                },
            ] -> null
          - size              = "s-1vcpu-2gb" -> null
          - tags              = [] -> null
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

digitalocean_kubernetes_cluster.cluster: Destroying... [id=c8beca30-502d-4d4d-807c-256bedff2e91]
digitalocean_kubernetes_cluster.cluster: Destruction complete after 1s

Destroy complete! Resources: 1 destroyed.

Terraform Configuration Files

# main.tf
terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

variable "do_token" {}

provider "digitalocean" {
  token = var.do_token
}

# do-secrets.tfvars
do_token = "your_do_token_12345"

Terraform version

Terraform v1.4.6
on darwin_arm64
+ provider registry.terraform.io/digitalocean/digitalocean v2.28.1

Debug Output

Panic Output

Additional context

• In resource_kubernetes_cluster.go, the function resourceDigitalOceanKubernetesClusterDelete() uses client.Kubernetes.Delete() (link)`, which corresponds to this API endpoint. The reference says that this endpoint is used to delete a Kubernetes cluster and all services deployed to it.

Important Factoids

References

[uniglot]

Can someone confirm whether this is an expected behavior?

[maartenpaauw]

I assumed terraform destroy would also destroy the Droplets created via the required node_pool configuration as well.

[andrewsomething]

Issuing a terraform destroy should indeed destroy the Droplets associated with the cluster's node pools. Though Terraform does not manage those Droplets directly. The destroy command makes an API request to the DigitalOcean Kubernetes Service's (DOKS) API. DOKS should then destroy the Droplets that are part of the cluster. This is an asynchronous operation that may take some time to complete.

Are these Droplets being left indefinitely or do they eventually go away?

[uniglot]

@andrewsomething Let me try again in a few days and then I can tell you whether the Droplets eventually get destroyed.

[miry]

I came to the same problem. My expectation that terraform destroy waits for node pool finish clear all droplets before exit.

In my case the droplet was there after 1 minute of kubernetes destroyed.