terraform-provider-vcd icon indicating copy to clipboard operation
terraform-provider-vcd copied to clipboard

Published 20 hours ago • verified publisher icon vmware

Sometimes vApp network creation or destroy fails with busy on VDC_UPDATE_VAPP_NETWORK_SECTION

Open lvirbalas opened this issue 5 years ago • 3 comments

Terraform Version

Terraform v0.11.7

Affected Resource(s)

  • vcd_vapp
  • vcd_vapp_vm
  • vcd_vapp_network

Terraform Configuration Files

# Configure the VMware vCloud Director Provider
provider "vcd" {
  url      = "https://${var.vcd_host}/api"
  org      = "Solpan"
  vdc      = "SolpanVDC"
  user     = "me"
  password = "${var.org_pass}"

  max_retry_timeout    = "300"
  allow_unverified_ssl = "true"

  logging = "true"
}

# Independent disk
resource "vcd_independent_disk" "TerraformDisk" {
  name         = "tf-disk"
  size         = "1024"        # MB
  bus_type     = "SCSI"
  bus_sub_type = "VirtualSCSI"
}

# Org VDC network
resource "vcd_network_routed" "TerraformNetwork" {
  name         = "TerraformNetwork"
  edge_gateway = "EdgeToVMware"

  gateway = "192.168.0.1"

  static_ip_pool {
    start_address = "192.168.0.2"
    end_address   = "192.168.0.100"
  }

  dhcp_pool {
    start_address = "192.168.0.101"
    end_address   = "192.168.0.200"
  }
}

# vApp in the routed network
resource "vcd_vapp" "TerraformVApp" {
  name = "TerraformVApp"

  depends_on = ["vcd_network_routed.TerraformNetwork"]
}

# vApp network
resource "vcd_vapp_network" "TerraformVAppNet" {
  name               = "tf-vapp-net"
  vapp_name          = "TerraformVApp"
  gateway            = "192.168.2.1"
  netmask            = "255.255.255.0"
  dns1               = "192.168.2.1"
  dns2               = "192.168.2.2"
  dns_suffix         = "test.org"
  guest_vlan_allowed = true

  static_ip_pool {
    start_address = "192.168.2.51"
    end_address   = "192.168.2.100"
  }

  dhcp_pool {
    start_address = "192.168.2.2"
    end_address   = "192.168.2.50"
  }

  depends_on = ["vcd_vapp.TerraformVApp"]
}

# vApp's VM connected to a network with routed connection to the outside
resource "vcd_vapp_vm" "TerraformVM7" {
  vapp_name     = "TerraformVApp"
  name          = "TerraformVM7"
  catalog_name  = "OperatingSystems"
  template_name = "Linux"
  memory        = 384
  cpus          = 4

  network_name = "TerraformNetwork"
  ip           = "192.168.0.7"

  disk {
    name        = "${vcd_independent_disk.TerraformDisk.name}"
    bus_number  = 1
    unit_number = 0
  }

  accept_all_eulas = "true"
  depends_on       = [/*"vcd_catalog_item.OVA",*/ "vcd_network_routed.TerraformNetwork", "vcd_vapp.TerraformVApp", "vcd_independent_disk.TerraformDisk"]
}

# Attach ISO to VM
resource "vcd_inserted_media" "TerraformInsertedISO" {
  catalog = "OperatingSystems"
  name    = "dsl-4.11.rc2.iso"

  vapp_name = "TerraformVApp"
  vm_name   = "TerraformVM7"

  depends_on = [/*"vcd_catalog_media.ds-linux",*/ "vcd_vapp_vm.TerraformVM7"]
}

resource "vcd_vapp_vm" "TerraformVM9" {
  vapp_name     = "TerraformVApp"
  name          = "TerraformVM9"
  catalog_name  = "OperatingSystems"
  template_name = "Linux"
  memory        = 384
  cpus          = 1

  #network_name = "TerraformNetwork"
  #ip           = "192.168.0.9"

  vapp_network_name = "tf-vapp-net"
  ip                = "192.168.2.9"
  cpus              = "4"
  cpu_cores         = "2"
  accept_all_eulas  = "true"
  depends_on        = [/*"vcd_catalog_item.OVA",*/ "vcd_vapp.TerraformVApp", "vcd_vapp_network.TerraformVAppNet"]
}

Steps to Reproduce

This one happens not always, but does happen often enough to get noticed. Best conditions for reproducing is to have a vApp with two VMs and a vApp network (you may leave this vApp network not connected to the VMs). Then just repeat terraform apply and terraform destroy until this error pops out.

Panic Output

This can happen during create:

vcd_vapp_network.TerraformVAppNet: Still creating... (10s elapsed)
vcd_vapp_vm.TerraformVM7: Still creating... (10s elapsed)
vcd_vapp_vm.TerraformVM7: Still creating... (20s elapsed)
...
vcd_vapp_vm.TerraformVM7: Still creating... (1m30s elapsed)
vcd_vapp_vm.TerraformVM7: Still creating... (1m40s elapsed)
vcd_vapp_vm.TerraformVM7: Creation complete after 1m42s (ID: TerraformVM7)
vcd_inserted_media.TerraformInsertedISO: Creating...
  catalog:     "" => "OperatingSystems"
  eject_force: "" => "true"
  name:        "" => "dsl-4.11.rc2.iso"
  vapp_name:   "" => "TerraformVApp"
  vm_name:     "" => "TerraformVM7"
vcd_inserted_media.TerraformInsertedISO: Still creating... (10s elapsed)
vcd_inserted_media.TerraformInsertedISO: Creation complete after 11s (ID: TerraformVApp_TerraformVM7_dsl-4.11.rc2.iso)

Error: Error applying plan:

1 error(s) occurred:

* vcd_vapp_network.TerraformVAppNet: 1 error(s) occurred:

* vcd_vapp_network.TerraformVAppNet: error creating vApp network. &errors.errorString{s:"error updating vApp Network: API Error: 400: [ 8d14e400-3f52-4108-acf8-ae2c52ac2299 ] The entity TerraformVApp is busy completing an operation VDC_UPDATE_VAPP_NETWORK_SECTION."}

It also can happen during destroy:

vcd_vapp_network.TerraformVAppNet: Destroying... (ID: tf-vapp-net)
vcd_firewall_rules.TerraformFWRule: Still destroying... (ID: EdgeToVMware, 10s elapsed)
vcd_inserted_media.TerraformInsertedISO: Still destroying... (ID: TerraformVApp_TerraformVM7_dsl-4.11.rc2.iso, 10s elapsed)
vcd_snat.outbound: Still destroying... (ID: 192.168.0.0/24, 10s elapsed)
vcd_dnat.sshVM: Still destroying... (ID: 10.150.211.101/32:2227 > 192.168.0.7/32:22, 10s elapsed)
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 10s elapsed)
vcd_dnat.sshVM: Destruction complete after 17s
vcd_firewall_rules.TerraformFWRule: Still destroying... (ID: EdgeToVMware, 20s elapsed)
vcd_inserted_media.TerraformInsertedISO: Still destroying... (ID: TerraformVApp_TerraformVM7_dsl-4.11.rc2.iso, 20s elapsed)
vcd_snat.outbound: Still destroying... (ID: 192.168.0.0/24, 20s elapsed)
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 20s elapsed)
vcd_inserted_media.TerraformInsertedISO: Destruction complete after 21s
vcd_vapp_vm.TerraformVM7: Destroying... (ID: TerraformVM7)
vcd_firewall_rules.TerraformFWRule: Still destroying... (ID: EdgeToVMware, 30s elapsed)
vcd_snat.outbound: Still destroying... (ID: 192.168.0.0/24, 30s elapsed)
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 30s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 10s elapsed)
vcd_firewall_rules.TerraformFWRule: Destruction complete after 31s
vcd_snat.outbound: Still destroying... (ID: 192.168.0.0/24, 40s elapsed)
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 40s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 20s elapsed)
vcd_snat.outbound: Destruction complete after 45s
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 50s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 30s elapsed)
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 1m0s elapsed)
...
vcd_vapp_vm.TerraformVM8: Still destroying... (ID: TerraformVM8, 3m0s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 2m40s elapsed)
vcd_vapp_vm.TerraformVM8: Destruction complete after 3m4s
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 2m50s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 3m0s elapsed)
...
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 4m50s elapsed)
vcd_vapp_vm.TerraformVM7: Still destroying... (ID: TerraformVM7, 5m0s elapsed)
vcd_vapp_vm.TerraformVM7: Destruction complete after 5m0s
vcd_independent_disk.TerraformDisk: Destroying... (ID: tf-disk)
vcd_independent_disk.TerraformDisk: Destruction complete after 6s

Error: Error applying plan:

1 error(s) occurred:

* vcd_vapp_network.TerraformVAppNet (destroy): 1 error(s) occurred:

* vcd_vapp_network.TerraformVAppNet: error removing vApp network: &errors.errorString{s:"error updating vApp Network: API Error: 400: [ 0f7a24d4-6f1b-4c86-a0ff-a3084b938649 ] The entity TerraformVM7 is busy completing an operation VDC_UPDATE_VAPP_NETWORK_SECTION."}

What is most interesting that the mentioned resource in the error message before "is busy" may not be using the vApp network at all - as in the example above, this TerraformVM7 is not using it.

Expected Behavior

No error.

Actual Behavior

The entity <vApp or VM> is busy completing an operation VDC_UPDATE_VAPP_NETWORK_SECTION

Important Factoids

Are we missing a wait for completion of the VDC_UPDATE_VAPP_NETWORK_SECTION operation when creating and destroying the vApp network?

lvirbalas avatar Mar 26 '19 08:03 lvirbalas

CC: @Didainius

lvirbalas avatar Mar 26 '19 08:03 lvirbalas

Having the exact same issue with: Terraform v0.12.4 & vCloud-Director v9.1. Is there anyway to extend the wait or timeout period to get past this limitation?

moortimis avatar Jul 17 '19 04:07 moortimis

I don't know @moortimis your case, but in this case as temporary solution should work adding (until issue fixed) additional depending:

# vApp network
# Attach ISO to VM
resource "vcd_inserted_media" "TerraformInsertedISO" {
   depends_on = ["vcd_vapp_vm.TerraformVM7", "vcd_vapp_network.TerraformVAppNet"]
}
  • making run insert media as last action

vbauzys avatar Jul 17 '19 04:07 vbauzys