Requested disk iops 0 for virtual machine "vm" exceeds maximum allowed iops 3,501

[xor222xor]

Hi there, After upgrading VMware Cloud Director to version: 10.2.0. Unable to create VM: Requested disk iops 0 for virtual machine "vm" exceeds maximum allowed iops 3,501

Version

Terraform v0.14.2 provider registry.terraform.io/vmware/vcd v3.0.0 VMware Cloud Director version: 10.2.0.17008054

Terraform resource

resource "vcd_vapp_vm" "vm" {
  vapp_name        = var.VAPP_NAME
  name                   = "vm"
  catalog_name     = var.VCD_CATALOG_NAME
  template_name    = var.VCD_TEMPLATE_VM
  memory                = var.RAM
  cpus                     = var.CPU
  storage_profile  = var.STORAGE_POLICY
}

Override_template_disk parameter can be applied after VM creation. Can I somehow set the iops value when creating the vm?

[Didainius]

Hello @xor222xor , As a first question do you have IOPS limits set anywhere in VCD (at provider VDC or Org VDC)?

[ghost]

Hello, @Didainius yes, on storage policies :

• IOPS Limiting Enabled: true
• Maximum Disk IOPS: 3501
• Default Disk IOPS: 3500
• Disk IOPS Per GB Max: 0
• IOPS Limit: 0
• Impact Placement: true

[Didainius]

Thanks, Could you try to use override_template_disk to alter IOPS. This is not ideal, but we want to understand if it helps. https://registry.terraform.io/providers/vmware/vcd/latest/docs/resources/vapp_vm#override-template-disk

[ghost]

I am trying to apply override_template_disk. But I get the error: Error: [VM creation] error adding VM: error instantiating a new VM: API Error: 400: [ 5e15490c-2f85-4d07-8130-aafefa370e63 ] Requested disk iops 0 for virtual machine "vm" exceeds maximum allowed iops 3,501

Probably, change is possible only after creation

Resource:

resource "vcd_vapp_vm" "vm" {
  vapp_name        = var.VAPP_NAME
  name                   = "vm"
  catalog_name     = var.VCD_CATALOG_NAME
  template_name    = var.VCD_TEMPLATE_VM
  memory                = 2000
  cpus                     = 2
  storage_profile  = var.STORAGE_POLICY

  override_template_disk {
    bus_type         = "paravirtual"
    size_in_mb       = "22384"
    bus_number       = 0
    unit_number      = 0
    iops             = "3500"
    storage_profile  = var.STORAGE_POLICY
  }
}

[Didainius]

@xor222xor , Thanks for info. Would you be able to capture log file https://registry.terraform.io/providers/vmware/vcd/latest/docs#logging

I have tried to replicate it applying the same limits as yours on pVdc storage policies and inheriting them in Org Vdc. Then I specified the storage profile.

My template has only 1 disk. Maybe your have more? Any other specifics?

resource "vcd_vapp_vm" "TestAccVcdVAppVmVm" {
  vapp_name     = vcd_vapp.TestAccVcdVAppVmVapp.name
  name          = "TestAccVcdVAppVmVm"
  catalog_name  = "cat-datacloud"
  template_name = "photon-hw11"
  memory        = 1024
  cpus          = 2

  storage_profile = "Development" # The one with limits as yours.
}

Update. For reference - the version I used is 10.2.0.17008054

[ghost]

@Didainius, There are no special features, the organization has two Storage Policies. The second one with no set limits on iops is no problem not when creating a VM or override_template_disk.

What is the best way to send api log to you?

[Didainius]

What is the best way to send api log to you?

The logs should have credentials obfuscated. You could probably replace hostnames in logs if you want to obfuscate and even putting into gist works https://gist.github.com/

[ghost]

@Didainius, ok https://raw.githubusercontent.com/xor222xor/vcd-test/main/terraform.log https://raw.githubusercontent.com/xor222xor/vcd-test/main/api.log https://raw.githubusercontent.com/xor222xor/vcd-test/main/main.tf

[Didainius]

Very strange. I still can't replicate it. I am still wondering if this is something related to image itself. Would you be able to give a try with this dead simple photon image - https://packages.vmware.com/photon/3.0/Rev3/ova/photon-hw11-3.0-a383732.ova

I want to understand where is the difference here.

[Didainius]

Also - one more question - does it work if you use UI to deploy?

[ghost]

Hello @Didainius,

Creation of vm with this image also failed. Yes, I can create a vm in Cloud Director ui without any problems. Also the config works well with storage without limits in this vcd_org

Addition: Before upgrading the VMware Cloud Director version, a similar iops error appeared when creating vcd_vm_internal_disk, but it was solved by specifying a value.

If you're interested, we could arrange access to vcd_org. Thank you.

https://raw.githubusercontent.com/xor222xor/vcd-test/photon-image/api.log https://raw.githubusercontent.com/xor222xor/vcd-test/photon-image/main.tf https://raw.githubusercontent.com/xor222xor/vcd-test/photon-image/terraform.log

[Didainius]

Would you be able to join Slack space https://code.vmware.com/join? We have a channel vcd-terraform-dev there

[ghost]

@Didainius The problem was reproduced in vcd ui. When creating a vm in vapp. Most likely this is not a problem of the provider. Thank you.

[Didainius]

@xor222xor , I will close this at the moment but would appreciate if you put an update once you get to the cause of it.

[jpbuecken]

Is it possible to reopen this issue, we see this issue in a VCD 10.1.1.16282995 (not our, we are a tenant there) but I will try to reproduce it in our own VCD 10.2.2.17855680. Did you try to reproduce the issue as provider user (system admin user in org context) or as tenant user (e.g. local org admin / user created in that org)?

[Didainius]

@jpbuecken , Some time has passed and it might need a refresher - do you have a simplified example of your exact case (even if it is for tenant in 10.1.1.16282995)?

[jpbuecken]

Hello, It took me the whole day, but I think I found the steps to reproduce.

First of all, my hcl

variable "hostname" {
  description = "Hostname for the instance"
  default     = "please_fill_me"
}

variable "image_catalog" {
  description = "Name of the catalog of the image/template"
  default     = "please_fill_me"
}

variable "image_string" {
  description = "Name of the image/template"
  default     = "please_fill_me"
}

variable "storage_profile" {
  description = "Set to use a special storage policy instead of the org vdc default"
  default     = null
}

resource "vcd_vapp" "vapp" {
  lifecycle {
    ignore_changes = [
      power_on,
    ]
  }
  name = var.hostname
}

resource "vcd_vapp_vm" "instance" {
  lifecycle {
    ignore_changes = [
      power_on,
    ]
  }
  vapp_name                = vcd_vapp.vapp.id == "always-not-equal" ? null : var.hostname
  name                     = var.hostname
  computer_name            = var.hostname
  catalog_name             = var.image_catalog
  template_name            = var.image_string
  cpus                     = 2
  cpu_cores                = 1
  memory                   = 4 * 1024
  power_on                 = false

  override_template_disk {
    bus_type        = "paravirtual"
    bus_number      = 0
    unit_number     = 0
    size_in_mb      = 30 * 1024
    storage_profile = var.storage_profile
  }
}

I tested it on 3 clouds (2 external where we are tenant only, and our own), so hostname, catalog name, catalog item and storage_profile needed to be variable.

The third one was a VCD 10.2.1.17444397, anyway I could reproduce it on our own VCD 10.2.2.17855680, so the version should not matter anymore:

1. Create a local user in the target org (not ldap) to authenticate. (VCD_SYS_ORG should not be set)
2. Give the local user the Organisation Administrator role.
3. ALL (!!!) storage policies attached to the VDC needs to have "IOPS Limiting Enabled = true". If only one storage policy has IOPS Limiting disabled, the issue is not triggered.

In my setup, I connected 2 policies, both with the same settings:

An allocation limit is not relevant.

It would be great if you can reproduce it now and figure out if you can workaround it inside the provider.

[Didainius]

I have tested it now (on VCD 10.2.2.17855680) and replicated.

With your sample HCL got error:

│ Error: error managing internal disks : error updating VM disks: error updating VM disks: API Error: 400: [ 9bb2519c-116e-4317-9b58-37c7116990b0 ] Requested disk iops 0 for virtual machine "testing" exceeds maximum allowed iops 7,001.
│ 
│   with vcd_vapp_vm.instance,
│   on main.tf line 47, in resource "vcd_vapp_vm" "instance":
│   47: resource "vcd_vapp_vm" "instance" {
│ 

In this API call:

2021/06/22 13:59:16 Request caller: govcd.(*VM).UpdateInternalDisksAsync-->govcd.(*VM).UpdateInternalDisksAsync-->govcd.executeRequestWithApiVersion-->govcd.(*Client).executeTaskRequest-->govcd.executeRequestCustomErr-->govcd.(*Client).NewRequestWithApiVersion-->govcd.(*Client).newRequest
2021/06/22 13:59:16 POST https://host/api/vApp/vm-7db0cbae-c993-4280-92eb-52260489588d/action/reconfigureVm
2021/06/22 13:59:16 --------------------------------------------------------------------------------
2021/06/22 13:59:16 Request data: [2446]
<?xml version="1.0" encoding="UTF-8"?>
  <Vm xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns="http://www.vmware.com/vcloud/v1.5" name="testing">
      <VmSpecSection Modified="true">
          <ovf:Info></ovf:Info>
          <OsType>other3xLinux64Guest</OsType>
          <NumCpus>1</NumCpus>
          <NumCoresPerSocket>1</NumCoresPerSocket>
          <CpuResourceMhz>
              <Configured>1</Configured>
          </CpuResourceMhz>
          <MemoryResourceMb>
              <Configured>2048</Configured>
          </MemoryResourceMb>
          <MediaSection>
              <MediaSettings>
                  <DeviceId>3002</DeviceId>
                  <MediaType>ISO</MediaType>
                  <MediaState>DISCONNECTED</MediaState>
                  <UnitNumber>0</UnitNumber>
                  <BusNumber>1</BusNumber>
                  <AdapterType>1</AdapterType>
              </MediaSettings>
              <MediaSettings>
                  <DeviceId>8000</DeviceId>
                  <MediaType>FLOPPY</MediaType>
                  <MediaState>DISCONNECTED</MediaState>
                  <UnitNumber>0</UnitNumber>
                  <BusNumber>0</BusNumber>
              </MediaSettings>
          </MediaSection>
          <DiskSection>
              <DiskSettings>
                  <DiskId>2000</DiskId>
                  <SizeMb>30720</SizeMb>
                  <UnitNumber>0</UnitNumber>
                  <BusNumber>0</BusNumber>
                  <AdapterType>5</AdapterType>
                  <ThinProvisioned>true</ThinProvisioned>
                  <StorageProfile href="https://host/api/vdcStorageProfile/2cfdae00-c147-46ba-b24a-a2e2e2252f8e" id="urn:vcloud:vdcstorageProfile:2cfdae00-c147-46ba-b24a-a2e2e2252f8e" name="*"></StorageProfile>
                  <overrideVmDefault>true</overrideVmDefault>
                  <iops>0</iops>
                  <VirtualQuantityUnit>byte</VirtualQuantityUnit>
              </DiskSettings>
          </DiskSection>
          <HardwareVersion href="https://host/api/vdc/8f38fde4-0473-4fd9-a000-6efcc694f5c1/hwv/vmx-11" type="application/vnd.vmware.vcloud.virtualHardwareVersion+xml">vmx-11</HardwareVersion>
          <VmToolsVersion>2147483647</VmToolsVersion>
          <VirtualCpuType>VM64</VirtualCpuType>
          <TimeSyncWithHost>false</TimeSyncWithHost>
      </VmSpecSection>
  </Vm>

Sounds like the problem is sending <iops>0</iops>which most probably means "unlimited" in API. Will have a glance if there is an easy fix available.

[jpbuecken]

Sounds like the problem is sending 0which most probably means "unlimited" in API. Will have a glance if there is an easy fix available.

It looks like a problem with "update" / reconfigureVM. Not with "create". Is the "iops" value included in the refresh? Maybe add the current value from state in the request?

[jpbuecken]

Another possible workaround for us would be https://github.com/vmware/terraform-provider-vcd/issues/692 If we can reference the "Default Disk IOPS" value we can just add it to our HCL.

e.g. like

data vcd_storage_profile "mypolicy" {
.....
}
.....

  override_template_disk {
    bus_type    = var.storage_controller
    bus_number  = 0
    unit_number = 0
    size_in_mb  = var.bootdisksizeinGB * 1024
    iops        = data.vcd_storage_profile.mypolicy.default_iops
  }

[Didainius]

I am closing this issue because the referenced solution PRs look to have been closed. Feel free to add extra data if this is still a problem.