terraform-provider-vsphere
terraform-provider-vsphere copied to clipboard
Published
20 hours ago •
hashicorp
hashicorp
`r/vapp_entity`: fails with: `...object 'vim.VirtualMachine:resgroup-xxxxxxx' has already been deleted or has not been completely created`
Terraform Version
v0.14.5
vSphere Provider Version
v1.24.3
Affected Resource(s)
vsphere_vapp_entity
Terraform Configuration Files
resource "vsphere_vapp_container" "vapp_container_cluster" {
name = var.cluster_name
parent_folder_id = data.vsphere_folder.folder.id
parent_resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
}
resource "vsphere_vapp_container" "vapp_container_controlplane" {
name = "${var.cluster_name}-controlplane"
parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id
parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}
resource "vsphere_vapp_container" "vapp_container_worker" {
name = "${var.cluster_name}-worker"
parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id
parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}
resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
target_id = vsphere_vapp_container.vapp_container_controlplane.id
container_id = vsphere_vapp_container.vapp_container_cluster.id
start_order = 1
start_action = "powerOn"
stop_action = "guestShutdown"
}
resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
target_id = vsphere_vapp_container.vapp_container_worker.id
container_id = vsphere_vapp_container.vapp_container_cluster.id
start_order = 2
start_action = "powerOn"
stop_action = "guestShutdown"
}
I have ommitted the other vsphere resources as they are all default stuff... i can add them if necessary :)
Debug Output
I'll add debug output after cleaning up all sensitive data
Panic Output
Expected Behavior
Settings for nested vApps should have been updated
Actual Behavior
Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415598' has already been deleted or has not been completely created
Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415599' has already been deleted or has not been completely created
Steps to Reproduce
created a nested vApp config like above with settings for
- start_order
- start_action
- stop_action
Important Factoids
References
- #0000
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Another thing i noticed.. sometimes if there were like 3 vms in a vApp, the settings where applied to one of them in the first run and to the others in the second run of terraform
Still a problem with
❯ terraform version
Terraform v0.15.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/vsphere v1.26.0
+ provider registry.terraform.io/rancher/rke v1.0.1
Can anyone verify the problems with nested vApps ?
Are you still seeing issue or was it resolved since originally posted?
Ryan
Thanks for the update @erSitzt. Happy to help if it's still a concern.
Ryan
Seems this is still not working...
❯ terraform version
Terraform v1.1.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/vsphere v2.0.2
On first run it fails with there errors:
vsphere_vapp_container.vapp_container_cluster: Creating...
vsphere_vapp_container.vapp_container_cluster: Creation complete after 1s [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creation complete after 0s [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Creating...
vsphere_vapp_container.vapp_container_worker: Creation complete after 1s [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappworker: Creating...
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│
│ with vsphere_vapp_entity.vapp_entity_vappcontrolplane,
│ on main.tf line 40, in resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane":
│ 40: resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
│
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│
│ with vsphere_vapp_entity.vapp_entity_vappworker,
│ on main.tf line 49, in resource "vsphere_vapp_entity" "vapp_entity_vappworker":
│ 49: resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
│
╵
Running apply again, does not fix it
❯ terraform apply
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
# vsphere_vapp_container.vapp_container_cluster has changed
~ resource "vsphere_vapp_container" "vapp_container_cluster" {
id = "resgroup-v522179"
name = "test-vapp-issue"
+ tags = []
# (12 unchanged attributes hidden)
}
# vsphere_vapp_container.vapp_container_controlplane has changed
~ resource "vsphere_vapp_container" "vapp_container_controlplane" {
id = "resgroup-v522180"
name = "test-controlplane"
+ tags = []
# (12 unchanged attributes hidden)
}
# vsphere_vapp_container.vapp_container_worker has changed
~ resource "vsphere_vapp_container" "vapp_container_worker" {
id = "resgroup-v522181"
name = "test-worker"
+ tags = []
# (12 unchanged attributes hidden)
}
# vsphere_vapp_entity.vapp_entity_vappcontrolplane has changed
~ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
id = "resgroup-v522179:resgroup-v522180"
~ stop_action = "guestShutdown" -> "powerOff"
# (7 unchanged attributes hidden)
}
# vsphere_vapp_entity.vapp_entity_vappworker has changed
~ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
id = "resgroup-v522179:resgroup-v522181"
~ stop_action = "guestShutdown" -> "powerOff"
# (7 unchanged attributes hidden)
}
Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# vsphere_vapp_entity.vapp_entity_vappcontrolplane is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
~ id = "resgroup-v522179:resgroup-v522180" -> (known after apply)
~ stop_action = "powerOff" -> "guestShutdown"
# (7 unchanged attributes hidden)
}
# vsphere_vapp_entity.vapp_entity_vappworker is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
~ id = "resgroup-v522179:resgroup-v522181" -> (known after apply)
~ stop_action = "powerOff" -> "guestShutdown"
# (7 unchanged attributes hidden)
}
Plan: 2 to add, 0 to change, 2 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│
│
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│
│
╵
And destroy will fail too...
❯ terraform destroy
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# vsphere_vapp_container.vapp_container_cluster will be destroyed
- resource "vsphere_vapp_container" "vapp_container_cluster" {
- cpu_expandable = true -> null
- cpu_limit = -1 -> null
- cpu_reservation = 0 -> null
- cpu_share_level = "normal" -> null
- cpu_shares = 4000 -> null
- id = "resgroup-v522179" -> null
- memory_expandable = true -> null
- memory_limit = -1 -> null
- memory_reservation = 0 -> null
- memory_share_level = "normal" -> null
- memory_shares = 163840 -> null
- name = "test-vapp-issue" -> null
- parent_folder_id = "group-v3" -> null
- parent_resource_pool_id = "resgroup-8" -> null
- tags = [] -> null
}
# vsphere_vapp_container.vapp_container_controlplane will be destroyed
- resource "vsphere_vapp_container" "vapp_container_controlplane" {
- cpu_expandable = true -> null
- cpu_limit = -1 -> null
- cpu_reservation = 0 -> null
- cpu_share_level = "normal" -> null
- cpu_shares = 4000 -> null
- id = "resgroup-v522180" -> null
- memory_expandable = true -> null
- memory_limit = -1 -> null
- memory_reservation = 0 -> null
- memory_share_level = "normal" -> null
- memory_shares = 163840 -> null
- name = "test-controlplane" -> null
- parent_folder_id = "resgroup-v522179" -> null
- parent_resource_pool_id = "resgroup-v522179" -> null
- tags = [] -> null
}
# vsphere_vapp_container.vapp_container_worker will be destroyed
- resource "vsphere_vapp_container" "vapp_container_worker" {
- cpu_expandable = true -> null
- cpu_limit = -1 -> null
- cpu_reservation = 0 -> null
- cpu_share_level = "normal" -> null
- cpu_shares = 4000 -> null
- id = "resgroup-v522181" -> null
- memory_expandable = true -> null
- memory_limit = -1 -> null
- memory_reservation = 0 -> null
- memory_share_level = "normal" -> null
- memory_shares = 163840 -> null
- name = "test-worker" -> null
- parent_folder_id = "resgroup-v522179" -> null
- parent_resource_pool_id = "resgroup-v522179" -> null
- tags = [] -> null
}
# vsphere_vapp_entity.vapp_entity_vappcontrolplane will be destroyed
- resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
- container_id = "resgroup-v522179" -> null
- id = "resgroup-v522179:resgroup-v522180" -> null
- start_action = "powerOn" -> null
- start_delay = 120 -> null
- start_order = 1 -> null
- stop_action = "powerOff" -> null
- stop_delay = 120 -> null
- target_id = "resgroup-v522180" -> null
- wait_for_guest = false -> null
}
# vsphere_vapp_entity.vapp_entity_vappworker will be destroyed
- resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
- container_id = "resgroup-v522179" -> null
- id = "resgroup-v522179:resgroup-v522181" -> null
- start_action = "powerOn" -> null
- start_delay = 120 -> null
- start_order = 2 -> null
- stop_action = "powerOff" -> null
- stop_delay = 120 -> null
- target_id = "resgroup-v522181" -> null
- wait_for_guest = false -> null
}
Plan: 0 to add, 0 to change, 5 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│
│
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│
│
╵
And this the state file after all this
terraform.tfstate
{
"version": 4,
"terraform_version": "1.1.5",
"serial": 9,
"lineage": "4f7eef30-b7a8-8da2-31ea-7375f11405c7",
"outputs": {},
"resources": [
{
"mode": "data",
"type": "vsphere_compute_cluster",
"name": "cluster",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"datacenter_id": "datacenter-2",
"id": "domain-c7",
"name": "Cluster1",
"resource_pool_id": "resgroup-8"
},
"sensitive_attributes": []
}
]
},
{
"mode": "data",
"type": "vsphere_datacenter",
"name": "dc",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "datacenter-2",
"name": "DC1"
},
"sensitive_attributes": []
}
]
},
{
"mode": "data",
"type": "vsphere_folder",
"name": "folder",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "group-v3",
"path": "/DC1/vm"
},
"sensitive_attributes": []
}
]
},
{
"mode": "managed",
"type": "vsphere_vapp_container",
"name": "vapp_container_cluster",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"cpu_expandable": true,
"cpu_limit": -1,
"cpu_reservation": 0,
"cpu_share_level": "normal",
"cpu_shares": 4000,
"custom_attributes": null,
"id": "resgroup-v522179",
"memory_expandable": true,
"memory_limit": -1,
"memory_reservation": 0,
"memory_share_level": "normal",
"memory_shares": 163840,
"name": "test-vapp-issue",
"parent_folder_id": "group-v3",
"parent_resource_pool_id": "resgroup-8",
"tags": []
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"data.vsphere_compute_cluster.cluster",
"data.vsphere_datacenter.dc",
"data.vsphere_folder.folder"
]
}
]
},
{
"mode": "managed",
"type": "vsphere_vapp_container",
"name": "vapp_container_controlplane",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"cpu_expandable": true,
"cpu_limit": -1,
"cpu_reservation": 0,
"cpu_share_level": "normal",
"cpu_shares": 4000,
"custom_attributes": null,
"id": "resgroup-v522180",
"memory_expandable": true,
"memory_limit": -1,
"memory_reservation": 0,
"memory_share_level": "normal",
"memory_shares": 163840,
"name": "test-controlplane",
"parent_folder_id": "resgroup-v522179",
"parent_resource_pool_id": "resgroup-v522179",
"tags": []
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"data.vsphere_compute_cluster.cluster",
"data.vsphere_datacenter.dc",
"data.vsphere_folder.folder",
"vsphere_vapp_container.vapp_container_cluster"
]
}
]
},
{
"mode": "managed",
"type": "vsphere_vapp_container",
"name": "vapp_container_worker",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"cpu_expandable": true,
"cpu_limit": -1,
"cpu_reservation": 0,
"cpu_share_level": "normal",
"cpu_shares": 4000,
"custom_attributes": null,
"id": "resgroup-v522181",
"memory_expandable": true,
"memory_limit": -1,
"memory_reservation": 0,
"memory_share_level": "normal",
"memory_shares": 163840,
"name": "test-worker",
"parent_folder_id": "resgroup-v522179",
"parent_resource_pool_id": "resgroup-v522179",
"tags": []
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"data.vsphere_compute_cluster.cluster",
"data.vsphere_datacenter.dc",
"data.vsphere_folder.folder",
"vsphere_vapp_container.vapp_container_cluster"
]
}
]
},
{
"mode": "managed",
"type": "vsphere_vapp_entity",
"name": "vapp_entity_vappcontrolplane",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"status": "tainted",
"schema_version": 0,
"attributes": {
"container_id": "resgroup-v522179",
"custom_attributes": null,
"id": "resgroup-v522179:resgroup-v522180",
"start_action": "powerOn",
"start_delay": 120,
"start_order": 1,
"stop_action": "powerOff",
"stop_delay": 120,
"tags": null,
"target_id": "resgroup-v522180",
"wait_for_guest": false
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"data.vsphere_compute_cluster.cluster",
"data.vsphere_datacenter.dc",
"data.vsphere_folder.folder",
"vsphere_vapp_container.vapp_container_cluster",
"vsphere_vapp_container.vapp_container_controlplane"
]
}
]
},
{
"mode": "managed",
"type": "vsphere_vapp_entity",
"name": "vapp_entity_vappworker",
"provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]",
"instances": [
{
"status": "tainted",
"schema_version": 0,
"attributes": {
"container_id": "resgroup-v522179",
"custom_attributes": null,
"id": "resgroup-v522179:resgroup-v522181",
"start_action": "powerOn",
"start_delay": 120,
"start_order": 2,
"stop_action": "powerOff",
"stop_delay": 120,
"tags": null,
"target_id": "resgroup-v522181",
"wait_for_guest": false
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"data.vsphere_compute_cluster.cluster",
"data.vsphere_datacenter.dc",
"data.vsphere_folder.folder",
"vsphere_vapp_container.vapp_container_cluster",
"vsphere_vapp_container.vapp_container_worker"
]
}
]
}
]
}
Hi @erSitzt,
Would you be able to share your example configuration files?
Ryan Johnson Staff II Solutions Architect | VMware, Inc.
Sure this is my test file
Click to expand!
provider "vsphere" {
user = "[email protected]"
password = "mypass"
vsphere_server = "vcenter.domain"
allow_unverified_ssl = true
}
data "vsphere_datacenter" "dc" {
name = "DC1"
}
data "vsphere_compute_cluster" "cluster" {
name = "Cluster1"
datacenter_id = data.vsphere_datacenter.dc.id
}
data "vsphere_folder" "folder" {
path = "/DC1/vm"
}
resource "vsphere_vapp_container" "vapp_container_cluster" {
name = "test-vapp-issue"
parent_folder_id = data.vsphere_folder.folder.id
parent_resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
}
resource "vsphere_vapp_container" "vapp_container_controlplane" {
name = "test-controlplane"
parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id
parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}
resource "vsphere_vapp_container" "vapp_container_worker" {
name = "test-worker"
parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id
parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}
resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
target_id = vsphere_vapp_container.vapp_container_controlplane.id
container_id = vsphere_vapp_container.vapp_container_cluster.id
start_order = 1
start_action = "powerOn"
stop_action = "guestShutdown"
}
resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
target_id = vsphere_vapp_container.vapp_container_worker.id
container_id = vsphere_vapp_container.vapp_container_cluster.id
start_order = 2
start_action = "powerOn"
stop_action = "guestShutdown"
}