Unable to deploy Kubevirt VM manifest

[evilhamsterman]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.10.4
Kubernetes provider version: 2.35.1
Kubernetes version: v1.31.4+k3s1

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"

}

resource "kubernetes_manifest" "vm_root" {
  wait {
    fields = {
      "status.phase" = "Succeeded"
    }
  }

  manifest = {
    apiVersion = "cdi.kubevirt.io/v1beta1"
    kind       = "DataVolume"
    metadata = {
      name      = "vm-test-root"
      namespace = "engineering"
    }
    spec = {
      storage = {
        accessModes = [
          "ReadWriteMany",
        ]
        resources = {
          requests = {
            storage = "200Gi"
          }
        }
      }
      source = {
        pvc = {
          namespace = "engineering"
          name      = "ubuntu-cloud-noble"
        }
      }
    }
  }
}


resource "kubernetes_manifest" "vm" {
  computed_fields = [
    "metadata",
    "status",
    "spec",
  ]
  wait {
    fields = {
      "status.ready" = true
    }
  }
  manifest = {
    "apiVersion" = "kubevirt.io/v1"
    "kind"       = "VirtualMachine"
    "metadata" = {
      "name"      = "vm-test"
      "namespace" = "engineering"
    }
    "spec" = {
      "runStrategy" = "Always"
      "template" = {
        "metadata" = {
          "labels" = {
            "role" = "user"
          }
        }
        "spec" = {
          "architecture" = "amd64"
          "domain" = {
            "cpu" = {
              "cores" = 8
              "model" = "host-passthrough"
            }
            "devices" = {
              "disks" = [
                {
                  "disk" = {
                    "bus" = "virtio"
                  }
                  "name"   = "root"
                  "serial" = "root"
                  "tag"    = "root"
                },
              ]
              "interfaces" = [
                {
                  "masquerade" = {}
                  "name"       = "default"
                },
              ]
            }
            "firmware" = {
              "uuid" = "9e6db865-b37b-df13-67c0-55ce5cc564b1"
            }
            "machine" = {
              "type" = "q35"
            }
            "memory" = {
              "guest" = "16Gi"
            }
            "resources" = {
              "requests" = {
                "memory" = "17Gi"
              }
            }
          }
          "networks" = [
            {
              "name" = "default"
              "pod" = {
                "vmNetworkCIDR" = "172.16.0.0/14"
              }
            },
          ]
          "terminationGracePeriodSeconds" = 180
          "volumes" = [
            {
              "dataVolume" = {
                "name" = kubernetes_manifest.vm_root.manifest.metadata.name
              }
              "name" = "root"
            },
          ]
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/evilhamsterman/adc5de1328dda90e9da31bfae7c0ac04

Panic Output

Steps to Reproduce

1. Install Kubernetes and Kubevirt
2. Create a terraform file with the kubernetes_manifest resource to try to create a Kubevirt VirtualMachine

Expected Behavior

It would apply successfully

Actual Behavior

Terraform successfully deploys everything including the VM, however it complains about inconsistent results for the VM and marks the run as failed

Important Factoids

I know about the computed_fields option, I've spent hours trying to different fields, even up to saying basically everything is computed but it still fails. It seems maybe something is not being read from the CRD correctly

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[refaelsa]

Having the same issue

kubernetes_manifest.virtual_machines["fedora-tf-vmi-res-1"]: Creating... ╷ │ Error: Provider produced inconsistent result after apply │ │ When applying changes to kubernetes_manifest.virtual_machines["fedora-tf-vmi-res-1"], provider "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an unexpected new value: .object: wrong final │ value type: incorrect object attributes. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker.

[schabrolles]

I’m in the same boat. Has anyone found a workaround?

[gugu]

What helped me:

1. I've deployed manifest with this error
2. kubectl get crd_type name -o yaml
3. converted yaml to terraform format and provided instead of given manifest

Looks like operator creates additional keys, which were not provided by original manifest

[EdwardCooke]

Figured something out that hopefully will help you with this. I had to add the creationTimestamp fields to the manifest in order for it to work. Here's a working example

resource "kubernetes_manifest" "virtualmachine" {
  manifest = {
    apiVersion = "kubevirt.io/v1"
    kind       = "VirtualMachine"
    metadata = {
      name      = "ubuntu-template-24-04"
      namespace = kubernetes_namespace.template.metadata[0].name
    }
    spec = {
      dataVolumeTemplates = [
        {
          metadata = {
            name = "ubuntu-24-04-template"
            creationTimestamp = null
          }
          spec = {
            storage = {
              resources = {
                requests = {
                  storage = "10Gi"
                }
              }
            }
            source = {
              pvc = {
                name      = "ubuntu-base"
                namespace = kubernetes_namespace.template.metadata[0].name
              }
            }
          }
        }
      ]
      runStrategy = "RerunOnFailure"
      template = {
        metadata = {
            creationTimestamp = null
        }
        spec = {
          architecture = "amd64"
          domain = {
            cpu = {
              cores   = 2
              sockets = 1
              threads = 1
            }
            devices = {
              disks = [
                {
                  name = "rootdisk"
                  disk = {
                    bus = "virtio"
                  }
                },
                {
                  name = "cloudinitdisk"
                  disk = {
                    bus = "virtio"
                  }
                  volumeName = "cloudinitdisk"
                }
              ]
              interfaces = [
                {
                  bridge = {}
                  model  = "virtio"
                  name   = "default"
                }
              ]
            }
            features = {
              acpi = {
                enabled = true
              }
            }
            machine = {
              type = "q35"
            }
            memory = {
              guest = "2Gi"
            }
            resources = {
              limits = {
                cpu    = "2"
                memory = "2Gi"
              }
              requests = {
                cpu    = "250m"
                memory = "1Gi"
              }
            }

          }
          evictionStrategy = "LiveMigrateIfPossible"
          hostname         = kubernetes_namespace.template.metadata[0].name
          networks = [
            {
              multus = {
                networkName = "ubuntu-template-24-04"
              }
              name = "default"
            }
          ]
          terminationGracePeriodSeconds = 30
          volumes = [
            {
              name = "rootdisk"
              dataVolume = {
                name = "ubuntu-24-04-template"
              }
            },
            {
              name = "cloudinitdisk"
              cloudInitNoCloud = {
                secretRef = {
                  name = kubernetes_secret.cloud-init.metadata[0].name
                }
                networkDataSecretRef = {
                  name = kubernetes_secret.cloud-init.metadata[0].name
                }
              }
            }
          ]
        }
      }
    }
  }
  computed_fields = [
    "metadata.annotations",
    "metadata.labels",
    "spec.dataVolumeTemplates[0].metadata.creationTimestamp",
    "spec.template.metadata.creationTimestamp",
    "spec.template.spec.domain.devices.interfaces[0].macAddress",
  ]
}